Detection, prediction, and classification for ocular disease

ABSTRACT

Computer systems and computer-implemented methods for performing classification, detection, and/or prediction based on processing of ocular images obtained from various imaging modalities are disclosed. Use of delayed near-infrared analysis (DNIRA) as one of the imaging modality is also disclosed.

FIELD

This generally relates to computer systems for ocular imaging anddisease evaluation.

BACKGROUND

Because ocular biopsy and the evaluation of systemic (e.g., non-ocular)samples such as blood or urine are not part of routine eye care,ophthalmologists and vision specialists rely on in vivo imaging for thediagnosis, classification and stratification of disease, patientmanagement, and clinical trial design.

However, for many potentially blinding eye diseases, it is presently notpossible to adequately characterize disease features or complexphenotypes using image-based biomarkers. As such, it is often notpossible to adequately diagnose, stage, grade, prognosticate, monitor orpredict response to treatments or interventions, or to predict safety ofthose treatments or interventions. New image-based, computer-basedanalytic tools offer such opportunities.

By way of non-limiting example, it is presently not possible to readilyidentify patients at imminent risk of progressing from early to lateblinding Age Related Macular Degeneration (AMD) so studies aimed toreduce the onset of geographic atrophy (GA) or choroidalneovascularization (CNV) (e.g., to progress from early to late disease)are not feasible. Further, because central visual acuity is not asuitable endpoint for studies of dry AMD, Phase III clinical trialdesign is limited to the enrollment of patients with pre-existentpatches of geographic atrophy (GA), typically of a defined size range,and prospectively measuring and comparing the rates of geographicatrophy expansion in study eyes and control eyes. Such endpoints addressdisease only after the late complications develop. Similarly by way ofnon-limiting example, it is presently not possible to readily predictpatients with pre-existent GA who are likely to expand quickly versusthose who will expand showly.

SUMMARY

There is a clear unmet need to develop better methods to classifypatients with blinding eye diseases using image-based biomarkers, bothqualitatively (e.g. describing new phenotypes) and quantitatively(measuring for example, the extent, quantity, en face 2-dimensional (2D)area, cross-sectional 2D area or 3-dimensional (3D) volume of diseasefeatures and the dynamic change in these features over time). There isalso a clear unmet need to develop methods to predict future changes indisease (e.g. progression for early to late, expansion of pre-existentGA) using image-based biomarkers, and improved classification schemes(phenotypes). Further, there is a clear unmet met to provide functionaltesting to evaluate the impact of disease and disease treatment onpatient outcomes. Unfortunately, most imaging is structural, while mostfunctional tests are both time-consuming and unreliable owing tovariable levels of effort and cognitive capacity of the individual beingtested. There is also clearly an unmet need to provide a functionalimaging method that relies on normal tissue physiology for theacquisition of the image. Such a functional imaging method may permitbetter biomarkers for clinical trial enrollment, outcome measure, andfor the evaluation of patient performance in response to such treatmentsin the clinical course of care.

Disclosed herein are computing systems for storage and analysis of aplurality of ocular images. In some embodiments, the computing systemcan comprise a computer processor. In some embodiments, the computingsystem can comprise a communications interface configured to import theplurality of ocular images, which plurality of ocular images correspondsto a plurality of different imaging modalities. In some embodiments, thecomputing system can comprise a database configured to store theplurality of ocular images. In some embodiments, the computing systemcan comprise a computer memory comprising a computer program. In someembodiments, the computer program can be executable by a computerprocessor to perform one or more of: (i) controlling the communicationsinterface to import the plurality of ocular images, (ii) processing theplurality of ocular images, and (iii) controlling the database to storethe plurality of ocular images. In some embodiments, the plurality ofdifferent imaging modalities can comprise delayed near-infrared analysis(DNIRA). In some embodiments, the plurality of different imagingmodalities can comprise one or more of, by way of non-limiting example:infra-red reflectance (IR), infra-red autofluorescence (IR-AF),near-infra-red autofluorescence (NIR-AF), confocal scanning laserophthalmoscopy (cSLO), fundus autofluorescence (FAF), color fundusphotography (CFP), optical coherence tomography (OCT), OCT-angiography(OCT-A), fluorescence life-time imaging ophthalmology or fluorescencelifetime imaging (FLI), and multispectral detection, and polarizedfluorescence imaging.

In some embodiments, the plurality of different imaging modalities cancomprise by way of non-limiting example, those able to detect ocularmanifestations of central nervous system (CNS) or brain diseases, forthe ocular detection of amyloid. In some embodiments, the plurality ofdifferent imaging modalities can comprise a functional modality. In someembodiments, the functional imaging modality can comprise, by way ofnon-limiting example, microperimetry, visual field testing,electroretinography (ERG), dark adaptometry (DA), or low luminancevisual acuity testing. In some embodiments, the plurality of ocularimages can comprise a plurality of different file formats. In someembodiments, the database can comprise a local storage unit. In someembodiments, the database can comprise an external storage unitcommunicatively coupled to the computer processor over a computernetwork. In some embodiments, the plurality of ocular images can beacquired from a subject during a single clinical visit. In someembodiments, the plurality of ocular images can be acquired from asubject during a plurality of clinical visits. In some embodiments, theplurality of ocular images can be acquired from a plurality of subjects.In some embodiments, the processing can comprise generating imagemetadata for each of the plurality of ocular images. In someembodiments, the image metadata can comprise a time stamp and a tag forfuture reference. In some embodiments, the processing can comprisegenerating composite images from the plurality of ocular images. In someembodiments, the processing can comprise generating pseudo-compositeimages from the plurality of ocular images. In some embodiments, thecomputer program can comprise a graphical user interface (GUI)configured to allow user selection of a plurality of analytical modules.In some embodiments, the plurality of analytical modules can performimage analysis on the plurality of ocular images. In some embodiments,the plurality of analytical modules can control the database to storethe plurality of ocular images. In some embodiments, the plurality ofanalytical modules can comprise a preprocessing module configured tonormalize and register the plurality of ocular images, therebygenerating a plurality of preprocessed ocular images. In someembodiments, the plurality of analytical modules can comprise asegmentation module configured to perform segmentation analysis of theplurality of preprocessed ocular images. In some embodiments, thesegmentation analysis can be based on one or more features selected fromthe group consisting of, by way of non-limiting example: circularity,granularity, area, location relative to fovea, location relative tooptic nerve head, location relative to periphery, number, clustering,thickness, degree of hyperfluorescence or hypofluorescence, degree ofhypo-reflectance or hypo-reflectance, softness or hardness of edges,confluence between or joining of individual features, and presence orabsence of known features in other modalities. In some embodiments, theknown features in other modalities can comprise, by way of non-limitingexample, features selected from the group consisting of: blood or bloodvessels, fluid, small hard drusen, medium drusen, large drusen,confluent drusen, serous, drusenoid or hemorrhagic RPE detachments,regions of geographic atrophy, active or quiescent choroidalneovascularization, nascent geographic atrophy, geographic atrophy,collapsed pigment epithelial detachment, basal laminar deposits, basallinear deposits, subretinal hyper-reflective material, hyper-reflectivematerial, hypo-fluorescent FAF, hyper-fluorescent FAF, hypo-reflectiveIR or hyper-reflective IR-AF, hypo-fluorescent or hyper-fluorescentNIR-AF, and scarring. In some embodiments, the segmentation analysis canbe mapped and/or masked on ocular images of a subject acquired atsubsequent clinical visits. In some embodiments, the segmentationanalysis can be used to quantify changes in size in ocular images of asubject acquired at subsequent clinical visits. In some embodiments, thesegmentation analysis can comprise storing the segmented ocular imagesin the database. In some embodiments, the segmentation analysis cancomprise displaying the segmented ocular images. In some embodiments,the plurality of analytical modules can comprise a delta moduleconfigured to perform a delta analysis of the plurality of preprocessedocular images. In some embodiments, the delta analysis can comprisedetermining changes between preprocessed ocular images acquired from asingle subject during different clinical visits. In some embodiments,the delta analysis can comprise determining changes between preprocessedocular images acquired from a single subject corresponding to aplurality of different imaging modalities. In some embodiments, thedelta analysis can comprise generating subtracted ocular imagesrepresentative of changes between preprocessed ocular images. In someembodiments, the delta analysis can comprise storing the subtractedocular images in the database. In some embodiments, the delta analysiscan comprise displaying the subtracted ocular images. In someembodiments, the subtracted ocular images can be displayed as an overlayon preprocessed ocular images acquired at earlier clinical visits. Insome embodiments, the plurality of analytical modules can comprise abroad image analytics module configured to perform a broad imageanalysis of the plurality of preprocessed ocular images, and theplurality of ocular images can be acquired from a plurality of subjects.In some embodiments, the broad image analysis can comprise aggregatingquantitative features extracted from the plurality of preprocessedocular images. In some embodiments, the aggregated quantitative featurescan be used to identify and quantify a burden of a degenerative eyedisease. In some embodiments, the aggregated quantitative features canbe used to identify, characterize, treat or prevent the onset ofreversible or irreversible eye tissue loss caused by an eye disease. Insome embodiments, the prevention can comprise performing an ocular orsystemic (non-ocular) administration of one or more of: a delivery ofgene therapy, a cell-based therapy, a laser-based therapy, and apharmacological or biological therapy. In some embodiments, theaggregated quantitative features can be processed by a cognitivetechnology configured to perform one or more of: (i) cognitive analysisof phenotyping, genotyping, and/or epigenetics, (ii) biomarkeridentification, (iii) estimating a probability of an eye disease, (iv)estimating a probability of effectiveness of a treatment for an eyedisease, (v) estimating a probability for recommending a treatment foran eye disease, and (vi) estimating a probability for recommending aclinical trial enrollment, (vii) estimating the likelihood of respondingto a treatment for an eye disease, (viii) estimating the safety of atreatment of an eye disease.

In some embodiments, the plurality of analytical modules can comprise ananalysis of unstructured data, configured to perform analysis of theplurality of ocular images. In some embodiments, the analysis ofunstructured data, or non-segmentable data, can be based on selected andunselected features. These may include, by way of non-limiting example:circularity, granularity, area, location relative to fovea, locationrelative to optic nerve head, location relative to periphery, number,clustering, thickness, degree of hyperfluorescence or hypofluorescence,degree of hypo-reflectance or hypo-reflectance, softness or hardness ofedges, confluence between or joining of individual features, complex enface patterns, arrays, distributions, and the presence or absence ofknown features in other modalities. Methods may include patternrecognition, signal processing, statistical analysis, deep learning,cognitive computing, convolutional neural network (CNN) and artificialintelligence. In some embodiments, the computational image analysis issemi-automated, fully automated, supervised or unsupervised. In someembodiment it may with known features in other modalities can comprise,by way of non-limiting example, features selected from the groupconsisting of: blood or blood vessels, fluid, small hard drusen, mediumdrusen, large drusen, confluent drusen, serous, drusenoid or hemorrhagicRPE detachments, regions of geographic atrophy, active or quiescentchoroidal neovascularization, nascent geographic atrophy, geographicatrophy, collapsed pigment epithelial detachment, basal laminardeposits, basal linear deposits, subretinal hyper-reflective material,hyper-reflective material, hypo-fluorescent FAF, hyper-fluorescent FAF,hypo-reflective IR or hyper-reflective IR-AF, hypo-fluorescent orhyper-fluorescent NIR-AF, and scarring. In some embodiments, thesegmentation analysis can be mapped and/or masked on ocular images of asubject acquired at subsequent clinical visits. In some embodiments, thesegmentation analysis can be used to quantify changes in size in ocularimages of a subject acquired at subsequent clinical visits. In someembodiments, In some embodiment it may not align with known features inother modalities can comprise, by way of non-limiting example, featuresselected from the group consisting of: blood or blood vessels, activechoroidal neovascularization, non-exudative neovascularization,quiescent neovascularization, fluid, small hard drusen, medium drusen,large drusen, confluent drusen, serous, drusenoid or hemorrhagic RPEdetachments, regions of geographic atrophy, nascent geographic atrophy,RORA, geographic atrophy, collapsed pigment epithelial detachment, basallaminar deposits, basal linear deposits, subretinal hyper-reflectivematerial, hyper-reflective material, hypo-fluorescent FAF,hyper-fluorescent FAF, hypo-reflective IR or hyper-reflective IR-AF,hypo-fluorescent or hyper-fluorescent NIR-AF, and scarring. In someembodiments, the segmentation analysis can be mapped and/or masked onocular images of a subject acquired at subsequent clinical visits. Insome embodiments, the segmentation analysis can be used to quantifychanges in size of image based features in ocular images of a subjectacquired at subsequent clinical visits.

Also disclosed herein are computer-implemented methods for storing andanalyzing a plurality of ocular images. In some embodiments, the methodscan comprise importing, using a computer processor, a plurality ofocular images corresponding to a plurality of different imagingmodalities. In some embodiments, the methods can comprise processing,using the computer processor, the plurality of ocular images. In someembodiments, the methods can comprise storing, using the computerprocessor, the plurality of ocular images in a database. In someembodiments, the plurality of different imaging modalities can comprisedelayed near-infrared analysis (DNIRA). In some embodiments, theplurality of different imaging modalities can comprise, by way ofnon-limiting example, one or more of: infra-red reflectance (IR),infra-red autofluorescence (IF-AF), near-infra-red autofluorescence(NIR-AF), confocal scanning laser ophthalmoscopy (cSLO), fundusautofluorescence (FAF), color fundus photography (CFP), opticalcoherence tomography (OCT), and OCT-angiography (OCT-A), fluorescencelifetime imaging (FLI), multispectral detection, and polarizedfluorescence imaging. In some embodiments, the plurality of differentimaging modalities can comprise, by way of non-limiting example, thoseable to detect ocular manifestations of central nervous system (CNS) orbrain diseases, for the ocular detection of amyloid. In someembodiments, the plurality of different imaging modalities can comprisea functional modality. In some embodiments, the functional modality cancomprise, by way of non-limiting example, microperimetry, visual fieldtesting, electroretinography (ERG), or dark adaptometry (DA), or lowluminance visual acuity testing. In some embodiments, the plurality ofocular images can comprise a plurality of different file formats. Insome embodiments, the database can comprise a local storage unit. Insome embodiments, the database can comprise an external storage unitcommunicatively coupled to the computer processor over a computernetwork. In some embodiments, the plurality of ocular images can beacquired from a subject during a single clinical visit. In someembodiments, the plurality of ocular images can be acquired from asubject during a plurality of clinical visits. In some embodiments, theplurality of ocular images can be acquired from a plurality of subjects.In some embodiments, the processing can comprise generating imagemetadata for each of the plurality of ocular images. In someembodiments, the image metadata can comprise a time stamp and a tag forfuture reference. In some embodiments, the processing can comprisegenerating composite images from the plurality of ocular images. In someembodiments, the processing can comprise generating pseudo-compositeimages from the plurality of ocular images. In some embodiments, themethods can further comprise using a graphical user interface (GUI) toselect from a plurality of analytical processes. In some embodiments,the analytical processes can perform image analysis on the plurality ofocular images. In some embodiments, the analytical processes can controlthe database to store the plurality of ocular images. In someembodiments, the plurality of analytical processes can comprise apreprocessing, which preprocessing can comprise normalizing andregistering the plurality of ocular images, thereby generating aplurality of preprocessed ocular images. In some embodiments, theplurality of analytical processes can comprise a segmentation analysisof the plurality of preprocessed ocular images. In some embodiments, thesegmentation analysis can be based on one or more features selected fromthe group consisting of, by way of non-limiting example: circularity,granularity, area, location relative to fovea, location relative tooptic nerve head, location relative to periphery, number, clustering,thickness, degree of hyperfluorescence or hypofluorescence, degree ofhyper-reflectance or hypo-reflectance, softness or hardness of edges,confluence between or joining of individual features, and presence orabsence of known features in other modalities. In some embodiments, theknown features in other modalities can comprise features selected fromthe group consisting of: blood or blood vessels, fluid, small harddrusen, medium drusen, large drusen, confluent drusen, serous, drusenoidor hemorrhagic RPE detachments, regions of geographic atrophy, active orquiescent choroidal neovascularization, nascent geographic atrophy,geographic atrophy, collapsed pigment epithelial detachment, basallaminar deposits, basal linear deposits, subretinal hyper-reflectivematerial, hyper-reflective material, hypo-fluorescent FAF orhyper-fluorescent FAF, hypo-reflective IR or hyper-reflective IR,hypo-reflective IR or hyper-reflective NIR, and scarring. In someembodiments, the segmentation analysis can be mapped and/or masked onocular images of a subject acquired at subsequent clinical visits. Insome embodiments, the segmentation analysis can be used to quantifychanges in size in ocular images of a subject acquired at subsequentclinical visits. In some embodiments, the segmentation analysis cancomprise storing the segmented ocular images in the database. In someembodiments, the segmentation analysis can comprise displaying thesegmented ocular images. In some embodiments, the plurality ofanalytical processes can comprise a delta analysis of the plurality ofpreprocessed ocular images. In some embodiments, the delta analysis cancomprise determining changes between preprocessed ocular images acquiredfrom a single subject during different clinical visits. In someembodiments, the delta analysis can comprise determining changes betweenpreprocessed ocular images acquired from a single subject correspondingto a plurality of different imaging modalities. In some embodiments, thedelta analysis can comprise generating subtracted ocular imagesrepresentative of changes between preprocessed ocular images. In someembodiments, the delta analysis can comprise storing the subtractedocular images in the database. In some embodiments, the delta analysiscan comprise displaying the subtracted ocular images. In someembodiments, the methods can further comprise displaying the subtractedocular images as an overlay on preprocessed ocular images acquired atearlier clinical visits. In some embodiments, the plurality ofanalytical processes can comprise a broad image analysis of theplurality of preprocessed ocular images, and the plurality of ocularimages can be acquired from a plurality of subjects. In someembodiments, the broad image analysis can comprise aggregatingquantitative features extracted from the plurality of preprocessedocular images. In some embodiments, the methods can further compriseidentifying and quantifying burden of a degenerative eye disease usingthe aggregated quantitative features. In some embodiments, the methodscan further comprise preventing the onset of reversible or irreversibleeye tissue loss caused by an eye disease using the aggregatedquantitative features. In some embodiments, the prevention can compriseperforming an ocular or systemic (non-ocular) administration of one ormore of: a delivery of gene therapy, a cell-based therapy, and apharmacological or biological therapy. In some embodiments, the methodscan further comprise processing the aggregated quantitative features bya cognitive technology to perform one or more of: (i) cognitive analysisof phenotyping, genotyping, and/or epigenetics, (ii) biomarkeridentification, (iii) estimating a probability of an eye disease, (iv)estimating a probability of effectiveness of a treatment for an eyedisease, (v) estimating a probability for recommending a treatment foran eye disease, and (vi) estimating a probability for recommending aclinical trial enrollment, (vii) estimating the likelihood of respondingto a treatment for an eye disease, (viii) estimating the safety of atreatment of an eye disease.

Also disclosed herein are methods of detecting an ocular disease in asubject. In some embodiments, the methods can comprise administeringsystemically to the subject a fluorescent agent. In some embodiments,the fluorescent agent can be internalized by, or accumulated within, acell, cell layer or tissue layer from the subject having the oculardisease. In some embodiments, the internalized agent can produce afluorescent emission within the cell, cell layer or tissue layer,thereby producing an image data. In some embodiments, the methods cancomprise analyzing the fluorescent emission using an algorithm. In someembodiments, the algorithm can comprise detecting a pattern offluorescence from the image data from the subject. In some embodiments,the algorithm can comprise comparing the pattern of fluorescence fromthe image data from the subject to a pattern of fluorescence from acontrol subject, or a subject with a differing eye disease. In someembodiments, the algorithm can comprise detecting the ocular disease ifthe image data from the subject has a greater or lesser degree offluorescence than the image data from the control subject, or a subjectwith differing eye disease. In some embodiments, the cell or cell layercan be localized in ocular tissue of the subject. In some embodiments,the cell can be a phagocytic cell. In some embodiments, the cell can bean immune cell or antigen presenting cell, retinal pigment epithelialcell, or photoreceptor. In some embodiments, the cell can be amacrophage. In some embodiments, the cells can be a resident orcirculating macrophage. The macrophage can be an inflammatorymacrophage, perivascular macrophage, parenchymal macrophage, microglialcell, dendritic cell, or other antigen-presenting cell. In someembodiments, the internalized agent can produce a fluorescent emissionat least about 2 hours, 6 hours, 12 hours or 24 hours after theadministering. In some embodiments, the internalized agent can produce afluorescent emission less than about 7 days or 14 days after theadministering. In some embodiments, the pattern of fluorescence cancomprise hyperfluorescence. In some embodiments, the pattern offluorescence can comprise hypofluorescence. In some embodiments, thefluorescent agent can be a near-infrared dye. In some embodiments, thenear-infrared dye can be indocyanine green. In some embodiments, theadministering can be ocular, intraocular, intravitreal, suprachoroidal,pen-ocular, sub-tenons, intravenous, intraarterial, transdermal, oral,intranasal, intramuscular, subcutaneous, sublingual, buccal, orsuppository. In some embodiments, the ocular disease can be an oculardisorder selected from the group consisting of dry age-related maculardegeneration (AMD), wet AMD, reticular pseudodrusen, late onset retinaldegeneration, and any combination thereof. In some embodiments, theocular disease can be any affecting the retinal pigment epithelium(RPE), photoreceptors, macrophages and cells of the immune system. Insome embodiments, the ocular disease can be central serous retinopathy(CSR), adult vitelliform disease, uveitis, both primary and secondary tosystemic disease (e.g. by way of non-limiting example, sarcoid,rheumatoid disease, the arthritidities, etc.), the white dot syndromes(to include MEWDS (multiple evanescent white dot syndrome), serpiginouschoroidopathy, AMPPE (acute multifocal posterior placoidepitheliopathy), POHS (presumed ocular histoplasmosis), or SerpiginousChorioretinopathy. In some embodiments, the disease can be a maculopathyor cone dystrophy such as, by way or non-limiting example, Stargardtdisease. In some embodiments, the disease can be an inheriteddegenerative disease such as Retinitis Pigmentosa (RP). In someembodiments, the ocular disease can be an ocular melanoma, an oculartumor or an infiltrating tumor. In some embodiments, the algorithm canfurther comprise segmenting the image data based on one or more featuresto form a segmented image data. In some embodiments, the one or morefeatures can be selected from, by way of non-limiting example,circularity, granularity, area, location relative to fovea, locationrelative to optic nerve head, location relative to periphery, number,clustering, thickness, degree of hyperfluorescence or hypofluorescence,degree of hyper-reflectance or hypo-reflectance, softness or hardness ofedges, confluence between or joining of individual features, complex2-dimensional (2D) pattern, and presence or absence of known features inother modalities. In some embodiments, the one or more features cancomprise a hyperfluorescent dot. In some embodiments, the methods canfurther comprise comparing the segmented image data to a secondsegmented image data generated at a subsequent clinical visit. In someembodiments, the methods can further comprise comparing the segmentedimage data to a second segmented image data generated at a priorclinical visit. In some embodiments, the methods can further comprisecomparing the segmented image data to a patient risk factor, anadditional imaging modality, a functional modality, or an epigeneticfactor. In some embodiments, the methods can further comprise comparingthe segmented image data to a patient risk factor. The patient riskfactor can be selected from, by way of non-limiting example, drusen,pseudodrusen, RPE detachments, pigment, hyper-reflective material, basallaminar deposits, basal linear deposits, hypo-fluorescent FAF,hyper-fluorescent FAF, geographic atrophy, choroidal neovascularization,genetic single nucleotide polymorphisms (SNPs), copy number variants(CNV), genome wide associative studies (GWAS), exome sequencing, fullgenome sequencing, genomics, proteomics, transcriptomics, ocularbiomarkers, systemic (blood, urine, tissue) biomarkers, environmentalrisk factors, or any combination thereof. In some embodiments, themethods can further comprise comparing the segmented image data to anadditional (vs, “a second”) imaging modality. In some embodiments, thecomparing the segmented image data can comprise a delta analysis of thesegmented image data. In some embodiments, the delta analysis cancomprise determining changes between the segmented image data acquiredfrom the subject during different clinical visits. In some embodiments,the delta analysis can comprise determining changes between thesegmented image data acquired from the subject corresponding and aplurality of different imaging modalities. In some embodiments, thedelta analysis can comprise generating a subtracted ocular imagerepresentative of changes between segmented image data. In someembodiments, the second imaging modality can be selected, by way ofnon-limiting example, from infra-red reflectance (IR), infra-redautofluorescence (IR-AF), near-infra-red autofluorescence (NIR-AF),confocal scanning laser ophthalmoscopy (cSLO), fundus autofluorescence(FAF), color fundus photography (CFP), optical coherence tomography(OCT), and OCT-angiography (OCT-A). In some embodiments, the methods canfurther comprise comparing the segmented image data to a functionalmodality. The functional modality can be selected, by way ofnon-limiting example, from microperimetry, electroretinography (ERG),visual field testing, dark adaptometry (DA), and low luminance visualacuity testing. In some embodiments, the methods can further comprisecomparing the segmented image data to an epigenetic factor. Theepigenetic factor can be selected from smoking, age, body mass index,obesity index, dietary intake, dietary vitamin consumption, lipid panel,cholesterol levels, blood pressure, family history, concurrentmedications, or a pre-existing condition. In some embodiments, thealgorithm can further comprise a process for transforming image data fordisplay. In some embodiments, the methods can further comprisepreprocessing the image data by a registration and a normalization ofthe image data. In some embodiments, the methods can further comprisepost-processing the image data. In some embodiments, post processing thedata can comprise transforming the image data to generate an outputimage data. In some embodiments, post processing the data can comprisedisplaying changes in segments of the image data over time. In someembodiments, post processing the data can comprise generating agraphical user interface of visual elements representative of the imageoutput data. In some embodiments, the algorithm further comprisesdisease diagnosis, risk stratification, monitoring, prognosis, or aselection and prediction of a treatment and its response. In someembodiments, the imaging method can be used as a surrogate biomarker fordiagnosis, prognosis, disease progression, treatment selection andprediction of a treatment response or clinical trial design, or anycombination thereof. In some embodiments, the imaging method can be usedas a companion diagnostic for detection and treatment of the oculardisease or clinical trial enrollment or exclusion. In some embodiments,the methods can further comprise administering a treatment to thesubject. In some embodiments, the treatment can comprise administering adrug or pharmaceutically acceptable salt thereof to the subject. In someembodiments, the administering can be ocular, intraocular, pen-ocular,intravitreal, suprachoroidal, intravenous, intraarterial, transdermal,oral, intranasal, intramuscular, subcutaneous, sublingual, buccal, orsuppository. In some embodiments, the methods can further comprisecommunicating a result via a communication medium.

According to an aspect, there is provided computer system comprising: aprocessor; a memory in communication with the processor, the memorystoring instructions that, when executed by the processor cause theprocessor to: at a training phase, receive training data correspondingto a plurality of ocular images; perform feature extraction and featureselection to generate features based on the training data to build apattern recognition model; at a classification phase, receive aplurality of ocular images corresponding to a plurality of imagingmodalities; classify features of the plurality of ocular images usingthe pattern recognition model.

In some embodiments, the pattern recognition model is at least one of aconvolutional neural network, machine learning, decision trees, logisticregression, principal components analysis, naive Bayes model, supportvector machine model, and nearest neighbor model.

In some embodiments, the feature extraction generates a masked image ofdefined shapes.

In some embodiments, the feature selection is based on at least one offocality of the defined shapes, a number of focal points per unit areaof the defined shapes, and a square root of an area of the definedshapes.

In some embodiments, the features are defined by areas of black orhypofluorescence.

In some embodiments, the features are defined by areas ofhyperfluorescence.

In some embodiments, the plurality of ocular images of the training datacorrespond to a plurality of imaging modalities.

In some embodiments, the training phase further comprises building apattern recognition model for each of the plurality of imagingmodalities.

In some embodiments, the plurality of ocular images comprises across-section image.

In some embodiments, the plurality of ocular images comprises an en faceimage.

In some embodiments, the training phase further comprises registeringthe plurality of ocular images to a common coordinate system.

In some embodiments, the training phase further comprises cross-modalfusion of the plurality of ocular images to a common coordinate system.

In some embodiments, the plurality of imaging modalities comprises atleast one of delayed near-infrared analysis (DNIRA), infra-redreflectance (IR), confocal scanning laser ophthalmoscopy (cSLO), fundusautofluorescence (FAF), color fundus photography (CFP), opticalcoherence tomography (OCT), OCT-angiography, fluorescence lifetimeimaging (FLI).

In some embodiments, the memory stores further instructions that, whenexecuted by the processor cause the processor to: generate across-section segmentation map corresponding to an en face region of aneye, each segment of the cross-section segmentation map corresponding toa cross-section image at that region of the eye; classifying eachsegment of the cross-section segmentation map as a phenotype of one ofnormal, drusen, retinal pigment epithelium detachments (RPEDs),pseudodrusen geographic atrophy, macular atrophy, or neovascularizationbased at least in part on classification of the cross-section imagecorresponding to that segment using the pattern recognition model.

In some embodiments, the plurality of ocular images comprises multiplecross-section images corresponding to multiple time points and thememory stores further instructions that, when executed by the processorcause the processor to: generate, for each of the multiple time points,a cross-section segmentation map corresponding to an en face region ofan eye, each segment of the cross-section segmentation map correspondingto a cross-section image at that region of the eye; classify eachsegment of each cross-section segmentation map as a phenotype of tissuestate of one of normal, drusen, retinal pigment epithelium detachments(RPEDs), pseudodrusen geographic atrophy, macular atrophy, orneovascularization, based at least in part on classification of thecross-section image corresponding to that segment using the patternrecognition model; and generate a time series data model based on thecross-section segmentation map at each of the multiple time points.

In some embodiments, the time series data model is based at least inpart on identified changes in the cross-section segmentation maps overtime.

In some embodiments, the time series data model visualizes diseaseprogression.

In some embodiments, the time series data model is based at least inpart on elapsed time between the multiple time points.

In some embodiments, the features selected comprise phenotypes of a userassociated with the plurality of ocular images.

In some embodiments, the memory stores further instructions that, whenexecuted by the processor cause the processor to: correlate the featureswith stage or grade variants of binding eye disease including AgeRelated Macular Degeneration (AMD), monogenic eye disease, inherited eyedisease and inflammatory eye disease.

In some embodiments, the memory stores further instructions that, whenexecuted by the processor cause the processor to: correlate the featureswith stage or grade variants of central nervous system (brain) disease,such as dementia and Alzheimer's disease.

According to another aspect, there is provided a computer-implementedmethod for detecting a phenotype, comprising: receiving a plurality ofocular images corresponding to a plurality of imaging modalities;registering the plurality of ocular images to a common coordinatesystem; classifying features of each of the plurality of ocular imagesusing a pattern recognition model; identifying features of the image asone or more phenotypes.

In some embodiments, the pattern recognition model is a convolutionalneural network built based on training data corresponding to a pluralityof ocular images and feature extraction and feature selection isperformed to generate features from the training data.

In some embodiments, the feature extraction generates a greyscale imageof defined shapes.

In some embodiments, the defined shapes include at least one of leopardspots, loose weave, grey smudge, and fingerling potatoes.

In some embodiments, the computer-implemented method further comprisescorrelating one or more of the defined shapes with the presence ofphagocytic immune cells such as macrophages.

In some embodiments, the computer-implemented method further comprisesgenerating one or more descriptors of characteristics of the identifiedphenotype, such as location, size, quantity and colour.

According to a further aspect, there is provided a computer-implementedmethod for predicting tissue loss, comprising: receiving a plurality ofocular images corresponding to a plurality of imaging modalities;registering the plurality of ocular images to a common coordinatesystem; classifying features of each of the plurality of ocular imagesusing a pattern recognition model; predicting tissue loss based at leastin part on the features.

In some embodiments, the plurality of imaging modalities comprises atleast one of delayed near-infrared analysis (DNIRA), infra-redreflectance (IR), confocal scanning laser ophthalmoscopy (cSLO), fundusautofluorescence (FAF), color fundus photography (CFP), opticalcoherence tomography (OCT), OCT-angiography and fluorescence lifetimeimaging (FLI).

In some embodiments, the plurality of imaging modalities includescross-section images and en face images.

In some embodiments, the pattern recognition model is a convolutionalneural network built based on training data corresponding to a pluralityof ocular images and feature extraction and feature selection performedto generate features from the training data.

In some embodiments, the feature extraction generates a masked image ofdefined shapes.

In some embodiments, the feature selection comprises one or more of anarea of geographical atrophy, a square root of area of geographicalatrophy, focality of geographical atrophy, focality index ofgeographical atrophy, and rate of geographical atrophy expansion.

In some embodiments, the predicting tissue loss is based on time seriesforecasting to predict tissue loss based on a time series data model.

In some embodiments, the time series data model is generated based onmultiple cross-section segmentation maps generated for each of multipletime points and corresponding cross-section images, each of thecross-section segmentation maps corresponding to an en face region of aneye, and each segment of the cross-section segmentation mapcorresponding to a cross-section image at that region of the eyeclassified as a phenotype of one of pseudodrusen, normal, drusen, orgeographical atrophy, based at least in part on classification of thecross-section image corresponding to that segment using a convolutionalneural network.

In some embodiments, the features selected comprise phenotypes of a userassociated with the plurality of ocular images.

In some embodiments, the computer-implemented method further comprisesidentifying the phenotypes as risk factors by correlating the phenotypeswith a rate of tissue loss over time.

In some embodiments, the predicting tissue loss is based at least inpart on non-image based biomarker data.

In some embodiments, the non-image based biomarker data comprisescharacteristics of a user associated with the plurality of ocularimages, the characteristics including at least one of age, genetics,sex, smoker, diet, health parameters, concurrent illness and concurrentmedications and therapies.

In some embodiments, the predicting tissue loss comprises predicting arate of tissue loss.

In some embodiments, the predicting tissue loss comprises predictingwhether tissue loss has previously occurred.

In some embodiments, the predicting tissue loss comprises predictingwhether tissue loss will occur in the future.

In some embodiments, the predicting tissue loss comprises predictingregions of disease progression and rate of disease progress.

In some embodiments, the predicting tissue loss comprises predictingprogression from early to late dry Age Related Macular Degeneration(AMD).

In some embodiments, the computer-implemented method further comprisespredicting a response of a patient to an intervention based at least inpart on the features.

According to another aspect, there is provided a computer-implementedmethod for predicting neovascularization, comprising: receiving aplurality of ocular images corresponding to a plurality of imagingmodalities; registering the plurality of ocular images to a commoncoordinate system; classifying features of each of the plurality ofocular images using a pattern recognition model; predictingneovascularization based at least in part on the features.

In some embodiments, the plurality of imaging modalities comprises atleast one of delayed near-infrared analysis (DNIRA), infra-redreflectance (IR), confocal scanning laser ophthalmoscopy (cSLO), fundusautofluorescence (FAF), color fundus photography (CFP), opticalcoherence tomography (OCT), OCT-angiography and fluorescence lifetimeimaging (FLI).

In some embodiments, the plurality of imaging modalities includingcross-section images and en face images.

In some embodiments, the pattern recognition model is a convolutionalneural network built based on training data corresponding to a pluralityof ocular images and feature extraction and feature selection performedto generate features from the training data.

In some embodiments, the feature extraction generates a masked image ofdefined shapes.

In some embodiments, the feature selection comprises one or more of anare of intra-retinal, subretinal or sub-retinal pigment epithelium fluidusing OCT, or dye leakage using angiography.

In some embodiments, the predicting neovascularization is based on timeseries forecasting to predict tissue loss based on a time series datamodel.

In some embodiments, the time series data model is generated based onmultiple cross-section segmentation maps generated for each of multipletime points and corresponding cross-section images, each of thecross-section segmentation maps corresponding to an en face region of aneye, and each segment of the cross-section segmentation mapcorresponding to a cross-section image at that region of the eyeclassified as a phenotype of one of normal, drusen, retinal pigmentepithelium detachment, geographic atrophy, macular atrophy orneovascularization, based at least in part on classification of thecross-section image corresponding to that segment using a convolutionalneural network.

In some embodiments, the features selected comprise phenotypes of a userassociated with the plurality of ocular images.

In some embodiments, the computer-implemented method further comprisesidentifying the phenotypes as risk factors by correlating the phenotypeswith a rate of tissue loss over time.

In some embodiments, the predicting neovascularization is based at leastin part on non-image based biomarker data.

In some embodiments, the non-image based biomarker data comprisescharacteristics of a user associated with the plurality of ocularimages, the characteristics including at least one of age, genetics,sex, smoker, diet, health parameters, concurrent illness and concurrentmedications or therapies.

In some embodiments, the predicting neovascularization comprisespredicting a onset of neovascularizaton.

In some embodiments, the predicting neovascularization comprisespredicting whether neovascularization has previously occurred.

In some embodiments, the predicting neovascularization comprisespredicting whether neovascularization will occur in the future.

In some embodiments, the predicting neovascularization comprisespredicting regions of disease progression and rate of disease progress.

In some embodiments, the predicting neovascularization comprisespredicting progression from early to late dry Age Related MacularDegeneration (AMD).

In some embodiments, the computer-implemented method further comprisespredicting a response of a patient to an intervention based at least inpart on the features.

In accordance with another aspect, there is provided a compound, such asa macrophage modulator, in particular2-((1-Benzyl-1H-indazol-3-yl)methoxy)-2-methylpropanoic acid and itsderivatives and formulations, for use in treating Age Related MacularDegeneration selected using DNIRA-based features or phenotypes, madedistinguishable using a DNIRA based classifier or a DNIRA-basedPREDICTOR as described herein.

In accordance with another aspect, there is provided a compound, such asa macrophage modulator, in particular2-((1-Benzyl-1H-indazol-3-yl)methoxy)-2-methylpropanoic acid and itsderivatives and formulations, for use in treating Age Related MacularDegeneration, comprising assaying the eye with DNIRA-based features orphenotypes, made distinguishable using a DNIRA based classifier or aDNIRA-based PREDICTOR as described herein, and administering atherapeutically effective amount of a macrophage modulator to thepatient if a DNIRA-based feature or phenotype is present.

BRIEF DESCRIPTION OF THE DRAWINGS

Novel features of exemplary embodiments are set forth with particularityin the appended claims. A better understanding of the features andadvantages will be obtained by reference to the following detaileddescription that sets forth illustrative embodiments, in which theprinciples of the disclosed systems and methods are utilized, and theaccompanying drawings of which:

FIG. 1 is a schematic diagram of a system for processing, storage andretrieval of ocular images. Image acquisition is followed by cloud-basedimage analysis to generate a complex subject phenotype. Together withgenetic and epigenetic data, this subject phenotype may feed thedownstream steps of biomarker development, improved clinical trialdesign, drug development, new treatments and ultimately, the selectionof specific treatments for individualized medicine from a menu ofoptions. (QA=quality assurance).

FIG. 2 is a schematic diagram of a system for processing, storage andretrieval of ocular images. The system 100 connects to an imageacquisition device 106 to receive ocular images of one or moremodalities. The system 100 also connects to an external image storageunit 102 to access ocular images of one or more modalities. The system100 processes the images to generate output data. In particular, thesystem 100 generates graphical representations (of visual elements) ofthe output data for display on user device 104. The user device 104 canalso provide user configurations and parameters to system 100 to providefeedback and input data, as well as dynamically control the display toupdate the graphical representations based on timing and subject data.

FIG. 3 is a flow diagram of a method for processing, storage andretrieval of ocular images. System 100 includes a server 200, database202, client application 204, user devices 206, and imaging devices 208.

FIG. 4 is a schematic diagram of a system for processing steps,analytics, and the end-use of image data for drug development of newtreatments. The diagram depicts the flow of data from the clinicalsetting, through the computational steps, integration with othermodalities, including phenotypes, genome data and epigenetics, andultimately to an end user for treatment development.

FIG. 5 is a schematic diagram of a system for processing, storage andretrieval of ocular images. System 100 has a retinal image capture unit402, a display device 450, and a server 200. Server 200 includes varioushardware and software units, including, preprocessing unit 404,post-processing unit 408, and a broad analytics unit 408. Server 200stores ocular images received from retinal image capture unit 402.Server 200 integrates analysis from components 406 and 408. Server 200generates graphical representations of output data for display as partof GUI on display device 450. Server 200 receives commands,configurations and parameters from display device 450. Server 200displays processed images on display device 450.

FIG. 6 is a schematic diagram of a system for the key steps ofprocessing ocular DNIRA images. Raw image input 602 is provided to animage preprocessing unit 404 to generate qualified image data 608 by thesteps of registration 410 and normalization 414. Image post-processingunit 406 further transforms and processes the qualified image data 608using segmentation 424 and feature extraction 434. Delta unit 614generates delta images as output data. Outputs include image outputs 618and quantitative outputs 620.

FIG. 7 is a schematic diagram of a computing device to implement aspectsof systems for processing, storage and retrieval of ocular images.Depicted is system 100 (including server 200 or user device 104, 206)for processing, storage and retrieval of ocular images. The systemincludes processor 702, memory 704, I/O interface 706, and networkinterface 708.

FIG. 8 is a flow diagram of a method for processing, storage andretrieval of ocular images involving different components describedherein. The diagram shows analysis of 2 images simultaneously 802, 804including components of registration 410, which is improved throughcycles of modification until sufficient quality is reached, featureextraction 810 using algorithms developed for the uniquely identifieddisease-associated features 812, comparison of the 2 images 814, andclassification 816.

FIG. 9 depicts the importance of image registration in serial ophthalmicimaging and analysis. Registration allows to accommodate for slightchanges in alignment, tilt, skew, magnification and any other parametersthat may vary across each patient visit. Example is shown using FAFimages. Further this figure demonstrates the expansion of central GA onFAF, showing only regions where the tissue has died and the damage isirreversible.

FIG. 10 depicts segmentation and feature extraction of a DNIRA image. Inthe left panel, DNIRA identifies regions of profound hypofluorescence(central black region). In the middle panel identification of the regionof DNIRA hypofluorescence is shown. In the right panel segmentation isused to select regions of hypofluorescent DNIRA signal, enabling forfurther feature extraction and quantification.

FIGS. 11A-11D, depict representative DNIRA images of diseased retinas.Arrows indicate hypofluorescent regions. Yellow lines indicate theboundaries of hypofluorescent regions.

FIG. 12 depicts representative DNIRA images compared to correspondingIR, FAF and OCT images. The left panel is an IR, the middle is an FAFimage, and the right panel is a DNIRA image. The bottom panel shows thecorresponding OCT.

FIG. 13 depicts representative DNIRA images compared to correspondingIR, FAF and fundus photo. The top left is a fundus photo showing drusen,the bottom left the corresponding DNIRA image showing regions of blackwhere the drusen are. The right panel shows IR (top) and FAF (bottom)where drusen are typically mode difficult to detect.

FIG. 14 depicts a representative DNIRA image of a patient with centralGA observed on FAF (left). The DNIRA image (right) shows greater numberof hypofluorescent signals, a greater total area of hypofluorescence,and greater amount of perimeter/border of hypofluroescence. Thedifference between these two images represents the “delta” portion ofthe signal.

FIGS. 15A and 15B show representative output data as a visualrepresentation comparing DNIRA to lower image representing an RPEdetachment on OCT.

FIG. 16 shows image output data as a visual representation comparing IR,FAF and OCT with DNIRA.

FIGS. 17A and 17B depict representative image of quantification of totalhypofluorescent DNIRA area as a marker of progressive or dynamic change,and comparison of DNIRA signal across multiple timepoints/sessions toobserve changes associated with increasing or decreasing DNIRA signal.

FIG. 18 depicts a comparison of DNIRA vs colour photos & clinicalgrading of uveal melanoma examples.

FIGS. 19A-19J depicts the analysis of Tumor Associated Macrophages(TAMs) using DNIRA. DNIRA images were assembled into composites andcorrelated to other modalities including fundus imaging, FAF, and OCT.Regions of interest were selected and analyzed using imaging andquantification techniques, compared in patients with melanomas,indeterminate lesions or benign nevi. ANOVA and Bonferroni multiplecomparisons analysis of regional dot densities were analyzed acrossgroups.

FIG. 20 shows comparative retinal images obtained by IR, FAF, and DNIRAimaging to detect intermediate grey signal in the DNIRA image. UnlikeFAF, the DNIRA image reveals abnormal or “sick” RPE as a darker grey.

FIGS. 21A-21E depicts examples of patterns that have an interwoven,lacy, reticular or spot-like configuration. In some instances areobserved patterns that are more coarse (wider) in aspect or finer(tighter) in aspect. These patterns may be indicative of differentsubtypes of AMD and therefore different response to therapeutic options.

FIG. 22 depicts structural and functional aspects of DNIRA signal inRPE. Upper panel shows the relative intensity of DNIRA signal relativeto corresponding tissue change in the lower panel. In the lower panel,the normal RPE, Bruch's membrane (BrM) and choroidal vasculature isillustrated, along with commonly observed changes that occur in dry AMD.Hypofluorescence is observed in association with GA, RPE detachments,and drusen or other sub-RPE deposits (basal linear, basal laminardeposits). A mid-grey signal is illustrated in regions where RPE are intheir correct anatomical position relative to the choroidal bloodvessels but where cell metabolism is surmised to be abnormal.

FIG. 23 depicts structural and functional aspects of DNIRA signal inmacrophages. Relative intensity of DNIRA signal is shown in the upperpanel relative to corresponding tissue change in the lower panel.Hyperfluorescent dots (of a particular size, shape and motility) thatare associated with DNIRA signal are identified as phagocytic immunecells, namely macrophages. Green arrows indicate movement/uptake of ICGdye from choroidal circulation to retinal/RPE complex. In the lowerpanel, presence of phagocytic macrophages that have taken up dye isillustrated.

FIG. 24 depicts representative example of Total Hypofluorescent DNIRAAreas as a Marker of Disease and Disease Burden (Diagnostic orPrognostic Biomarker).

FIG. 25 depicts representative example of Rates of Change of TotalHypofluorescent DNIRA Areas as a Marker of Prognosis or Response to anIntervention.

FIGS. 26A and 26B depict representative examples of DNIRA Features toIdentify Early Disease and Different Phenotypes of Early Disease(Diagnostic Biomarker).

FIGS. 27A and 27B depict representative examples of DNIRA Features toDistinguish Diseases of Similar Clinical Appearance to Assist withDiagnostic Accuracy (Diagnostic Biomarker).

FIGS. 28A and 28B depict representative examples of DNIRA FeaturesIdentify Inherited Monogenic Disorders and Disease that May Not Yet BeDiagnosed (NYD).

FIG. 29 depicts a representative example of DNIRA Feature to MonitorProgression of Disease or the Effect of a Treatment Over Time(Monitoring Biomarker).

FIG. 30 depicts a representative example of DNIRA Feature to MonitorDisease Progression, Comparing Within and Across Modalities and OtherImaging Biomarkers Over Time (Monitoring and Prognostic Biomarker).

FIG. 31 depicts a representative example of DNIRA Feature as a Biomarkerto Identify Patients Likely to Progress to Later Disease (PrognosticBiomarker).

FIG. 32 depicts representative example of DNIRA Feature as a Biomarkerto Quantify Aspects of Disease Known to Predict Progression (PrognosticBiomarker).

FIG. 33 depicts representative example of Using DNIRA to Correlate withDisease Pathogenesis (Predictive Biomarker).

FIG. 34A-34D depict representative examples of DNIRA to Detect MultipleTypes of Hyperfluorescent Dot Signals (Predictive Biomarkers).

FIG. 35 depicts representative example of DNIRA to Detect Static andDynamic Hyperfluorescent Dot Signals (Predictive Biomarkers).

FIG. 36 depicts representative example of DNIRA to detect Regions ofTissue Damage and Macrophage Activity in Central Serous Retinopathy(Predictive, Prognostic Biomarkers).

FIG. 37 depicts representative example of DNIRA to detect TwoPopulations of Bright Hyperfluoerscent Dots in Diffuse-Tricking AMD(Diagnostic, Prognostic Biomarkers).

FIG. 38 depicts representative example of DNIRA to Detect ReticularPseudodrusen.

FIG. 39 depicts representative example of DNIRA applied to the analysisof ocular inflammatory disease. In this example of presumed AcuteMultifocal Posterior Placoid Epitheliopathy (AMPPE), DNIRA demonstratesabnormal RPE/outer retina layer uptake of dye.

FIG. 40 depicts the use of DNIRA as a marker of high risk AMD featuresto identify progression from early to late disease.

FIGS. 41-44 depict representative examples of a sequence of DNIRA imagesand OCT images of a patient's eyes obtained over four sessions.

FIG. 45 depicts a representative example of a DNIRA signal in black(arrow) where the tumour cells are blocking the flow of dye-labeledblood in the choroid.

FIG. 46A depicts a classifier exemplary of an embodiment.

FIG. 46B depicts a detector exemplary of an embodiment.

FIG. 46C depicts a predictor exemplary of an embodiment.

FIG. 47 depicts a data structure for storing multimodal retinal images,exemplary of an embodiment.

FIG. 48 depicts convolutional neural network architecture exemplary ofan embodiment.

FIG. 49 depicts operation of the classifier of FIG. 46A, exemplary of anembodiment.

FIG. 50 depicts a representative example plots of model loss and modelaccuracy on training and validation datasets over training epochs.

FIG. 51 depicts a representative example of an input and output of aclassifier, exemplary of an embodiment.

FIG. 52 depicts an example sequence of masked images from fourtimepoints.

FIG. 53 depicts image registration, exemplary of an embodiment.

FIG. 54 depicts a representative example of a classified cross-sectionsegmentation map.

FIG. 55 depicts a representative example of a classified cross-sectionsegmentation map registered to an en face FAF image.

FIG. 56 depicts a representative example of classified cross-sectionsegmentation maps, each associated with a time point.

FIG. 57 depicts a representative example of en face OCT using IR imagereference to determine position of cross-section.

FIG. 58 depicts multi-modal image registration for multiple sessions,exemplary of an embodiment.

FIG. 59 depicts OCT scans performed in the near IR channel, exemplary ofan embodiment.

FIG. 60 depicts multi-modal image registration for multiple sessions,using a DNIRA image as a reference, exemplary of an embodiment.

FIG. 61 depicts operation of the detector of FIG. 46B, exemplary of anembodiment.

FIG. 62 depicts a representative example of images taken of a particularsubject's eye over five sessions.

FIG. 63 depicts operation of the predictor of FIG. 46C, exemplary of anembodiment.

FIG. 64 depicts an embodiment of predictor to predict diseaseprogression.

FIG. 65 depicts a representative example of a graphical representationof using DNIRA to detect and quantify known risk factors for diseaseprogression.

FIG. 66 depicts a representative example multi-modal imaging of softdrusen.

FIG. 67 depicts a region of soft drusen as detected by an embodiment ofa detector.

FIGS. 68-69 depict representative examples of analysis of soft drusen bythe quantification of soft fuzzy drusen and RPEDs.

FIG. 70 depicts representative examples of multi-modal imaging of softdrusen.

FIG. 71 depicts a TAM as detected by an embodiment of a detector.

FIG. 72 depicts a representative example of a DNIRA image.

FIG. 73 depicts macrophages as detected by an embodiment of a detector.

FIG. 74 depicts a representative example of the application of K-meanssegmentation to a set of four DNIRA images.

FIG. 75 depicts an example 2D pattern, as observed in an FAF image, aDNIRA image, and a DNIRA image with regions classified by an embodimentof a classifier.

DETAILED DESCRIPTION Overview

Ocular diseases or disorders, which may reduce or eliminate one's sight,among other effects, are a major medical concern. For instance,age-related macular degeneration (AMD) is a leading cause of oculardysfunction, including irreversible blindness, especially in subjectsover 50 years old. All subjects start with early non-exudative,so-called, “dry” AMD. At the early stage, dry AMD is characterized bydeposits known as drusen, or in some cases pseudodrusen, that can beseen clinically in the posterior pole of the eye known as the macula.Advanced AMD can take two forms—late “wet” or late “dry”. The primarytreatment option for advanced “wet” AMD is regular intra-ocularinjections of antiangiogenic drugs. These injections are given afterpathological new blood vessels grow beneath or into the central retina,where they leak, bleed or cause tissue damage and scarring. By contrast,there are no treatments for early or late dry AMD. Late dry AMD ischaracterized by death or atrophy of the photoreceptors and theirassociated “nurse” cell layer, the retinal pigment epithelium (RPE).Patches of irreversible RPE and photoreceptor tissue loss are known as“geographic atrophy” (GA). GA can occur centrally, in the fovea, robbingsubjects of their central vision early in disease (measured as adecrease in visual acuity), or can accumulate extra- or juxta-foveally,leaving central visual acuity intact in the face of significant disease.These non-foveal scotoma, or blind spots, cause significant disabilityparticularly with respect to reading speed, tracking and text anddetecting or identifying non-central objects. This means that visualacuity (e.g., reading the eye chart) is not a suitable endpoint forclinical trial, nor a suitable measure for clinical trial enrichment.

Because ocular biopsy and the evaluation of systemic (e.g., non-ocular)samples such as blood or urine are not part of routine eye care,ophthalmologists and vision specialists rely on in vivo imaging for thediagnosis, classification and stratification of disease, patientmanagement, and clinical trial design. Fundus Autofluorescence (FAF) isa standard current method imaging dry AMD, and relies on thenon-invasive detection of endogenous fluorophores such as lipofuscinpigments within the RPE monolayer that, when stimulated with fluorescentlight in the blue spectrum, provide a diffuse homogenous ground-glassglow in the normal eye. In regions where RPE is lost (e.g., in regionsof GA) so too is the autofluorescent FAF signal. By virtue of theirprofoundly hypofluorescent, often black signal with sharp borders, the2-dimensional (2D) areas of GA can be readily quantified. These blacksignals contrast sharply against the remainder of the image whichexhibits variable shades of grey (grey-scale) and therefore cannot bereliably measured, both between patient visits and between patients.Newer methods of Optical Coherence Tomography (OCT), when reconstructedto form a slab, may be viewed en face to demonstrate regions of GA.However, at present, no imaging method other than FAF provides readilyquantifiable regions of profoundly hypofluorescent, or black, signal tocharacterize AMD or other ocular diseases or disorders.

It follows, in the absence of other suitable measures of disease, thatthe quantification of GA and its rate of expansion over time may be usedto drive clinical trial design, serving for example as inclusioncriterion or endpoint, respectively. However, the detection of GA by FAFidentifies areas of RPE and photoreceptor loss after tissue alreadymissing, and thus when it cannot be rescued or recovered. It does notprovide a measure of tissue at risk for future loss, provide anindication of tissue suitable for therapeutic protection, identify whomay be at imminent risk of developing GA (progressing from early to latedry AMD), nor does it provide insight into disease pathogenesis andsubtypes of disease. Therefore, there is a paucity of methods tocharacterize and quantify the features of early and late dry AMD.

Accordingly there is a need for new, or at least alternate, ophthalmicimaging methods for ocular diseases or disorders, including AMD. Suchmethods can be independent of patient fixation (which may be lost withcentral GA), may not rely on prolonged or intense activity (that oftenwanes dues to patient fatigue), and may not require significantcognitive ability (which can decline in the elderly, particularlyamongst patients with neurodegenerative disease). There is also a needfor new and alternative imaging techniques and a system for processing,storage and retrieval of ocular images from multiple imaging modalities.Additionally, there is a move for personalized medicine, image-guidedmedicine, and biomarker discovery, to accelerate/automate drugdiscovery, and optimize health care and medical solutions for patients.In this way, scarce resources can be better distributed during drugdevelopment stages, minimizing variability during clinical trials. To doso, there is an immense need for machine learning, deep learning,artificial intelligence, healthcare analytics and cloud computingsystems.

Disclosed herein are methods, systems and compositions that can beuseful for the diagnosis, treatment, or prevention of an ocular diseaseor disorder, including the use of in vivo imaging methods, imageprocessing methods, and generation of graphical representations usingthe output from the image processing, as part of an integrated platform.

Described herein is a novel functional imaging method, DelayedNear-Infrared Analysis (DNIRA) that has generated unprecedented imagesof the retina (the film of the eye) that include both known and unknownphenotypes of disease. These complex images can be compared against anindividual's genetic makeup (their genotype) and their concurrentillnesses, medications, and lifestyle history (their epigenetics). Thisimaging can be used as a diagnosis tool and to help identify particularsubjects that may benefit from certain clinical trials and drugdevelopment programs, and ultimately guiding clinical trial design andtreatment outcomes. In some aspects, when DNIRA is used specifically forimaging subjects with AMD, it is referred to as AMD Imaging (AMDI). Insome aspects, when DNIRA is used to image tumor associated macrophages(TAMs) in ocular melanoma subjects DNIRA is referred to as TumorAssociated Macrophage Imaging (TAMI). However, as the methodology hasutility in multiple ocular disorders, it can also be generally referredto by its broad, inclusive name, DNIRA.

Accordingly, described herein are methods for detecting an oculardisease or disorder or its late blinding complication(s), comprisingadministering to a subject in need thereof an effective amount of anagent capable of enhancing optical detection of the RPE, photoreceptorsor other component of the retina or associated tissue including cells ofthe immune system, based on the presence of endogenous fluorophores,chromophores or chemiluminescent compounds, or based on, but not limitedto, the physiological uptake, processing, activation or accumulation ofexogenous fluorphores, chromophores or chemiluminescent compounds thatmay be provided locally or systemically and differentially handled ormetabolized by those tissues.

In some aspects, described herein are methods of detecting and imagingthe physiological uptake, processing, activation or accumulation of asystemically administered dye, comprising administering to a subject asystemic dye such as indocyanine green (ICG) or its derivatives, andphotographing using specialized imaging equipment, its presence in thehours and days thereafter using appropriate excitation and emissionfilters or optical systems, without further administration of dye.

In some aspects, described herein are systems for processing the imagesobtained using such methods to generate graphical representations ofoutput data. In some aspects, described herein are systems that canprovide a logical, hardware and software unit for analyzing, quantifyingqualitatively representing the images obtained using such methods. Insome aspects, described herein are methods that provide an integratedplatform for image acquisition, processing, and output. The platform canstore and retrieve imaging data and generate graphical representationsof the imaging data. In some aspects, described herein are methods thatprovide a graphical user interface (GUI) with dynamic graphicalrepresentations of imaging data from the images obtained using themethods described herein. In some aspects, described herein are systemsprovide a centralized or networked image processing platform forprocessing, storage and retrieval of ocular images from multiple imagingmodalities and/or a cloud server.

In various aspects, systems described herein can relate to AMDI (AMDImaging), which is comprised of the clinical application of delayed nearinfrared analysis (DNIRA) along with the logical processes, hardware andsoftware units and an integrated platform for its analysis,interpretation, dissemination, application and integration with otherimaging and non-imaging modalities used to evaluate subjects with AMD orRPD, other potentially blinding diseases, and other ocular abnormalitieswhere macrophage are present.

In various aspects, systems described herein can relate to TAMI (TumorAssociated Macrophage Imaging), which is comprised of the clinicalapplication of DNIRA along with the logical processes, hardware andsoftware units and an integrated platform for its analysis,interpretation, dissemination, application and integration with otherimaging and non-imaging modalities used to evaluate subjects with ocularmelanoma, ocular tumors, and other ocular abnormalities wheremacrophages are present.

Aspects described herein further relate to the use of DNIRA/AMDI/TAMI asentrance inclusion or exclusion criteria, or endpoint analysis forclinical trial design. Some aspects can comprise the development andapplication of a complementary biomarker for clinical trial design anddrug development. Some aspects can comprise the use of DNIRA/AMDI/TAMIas a marker for diagnosis and/or prognosis and/or progression of anocular disease or disorder (including, without limitation AMD and RPD,Ocular Melanoma, Diabetic Retinopathy, Inherited Retinal Disease,Uveitis, and others).

Also described herein are methods and compositions that can be usefulfor the diagnosis, treatment, or prevention of an ocular disease ordisorder, including the use of in vivo imaging methods, their outputs,their processes, and management, as part of an integrated system forcomputer assisted analysis and diagnosis (CAAD). The system can use asinputs images of the eye (macula/posterior retina) from various imagingmodalities for comparative processing and provide as output graphicalrepresentations of a plurality of measures including prognostic anddiagnostic scores upon which to base personalized treatments fordiseases of the eye. The images may include standard clinical macularimages as well as the novel imaging types as described herein.

The system can use as input data images from multiple modalities anduser configurations or parameters. The system implements imageprocessing to obtain measurements and other reporting data for theimages. The system has one or more processing components to register theimages, segment objects of interest from the images, define patterns ofsegmented objects and/or intensity texture of the images. Theinformation can be aggregated by the system into a feature vector foreach subject. The subject's data can be used as input for statisticalmodels to generate diagnostic and prognostic scores which can be eithercategorical or continuous in nature. The system can rely on expert usersto provide training or configuration information for image processingand to the review the outputs from the image processing to provide afeedback loop to check the suitability of the results.

Also described herein are systems for image processing and displaycomprising: an image acquisition device to capture image data; datastorage to store and retrieve the image data; a processor configuredwith a preprocessing unit, a post-processing unit, and broad analyticsunit to transform the image data to generate output image data, the datastorage storing the output image data; and a client application toconnect to a display device to generate and update a graphical userinterface (GUI) of visual elements representative of the image outputdata. In some aspects, the preprocessing unit can be configured toregister and normalize the image data. In some aspects, thepreprocessing unit can be configured to register the image data usingregistration parameters that are dynamically adjusted using a feedbackloop of control commands to verify, approve and adjust the registrationparameters. In some aspects, the post-processing unit can be configuredto segment the image data and feature extract from the segmented imagedata. In some aspects, the broad analytics unit is configured tocomparatively process the image data to generate comparative image dataas a portion of the output image data.

In some aspects, a system described herein can have a delta image unitto cross reference and compare segments of the image data over a timeperiod, and across modalities to automatically monitor the image datafor changes to features of the segments of the image data. In someaspects, a first segment of the image data at a first time point can bemapped or masked on subsequent segments of the image data at subsequenttime points. In some aspects, the system can have a delta image unit tocompare a first image segment to a second image segment to generatedifference image data, and a client application to generate visualelements of the different image data to visualize a level of changebetween the image segments.

Also described herein are devices for image processing and displaycomprising: a data storage to store and retrieve image data; processorconfigured with a preprocessing unit, a post-processing unit, and broadanalytics unit to transform the image data to generate output imagedata, the data storage storing the output image data; and a clientapplication to connect to a display device to generate and update agraphical user interface of visual elements representative of the imageoutput data. In some aspects, the preprocessing unit is configured toregister and normalize the image data. In some aspects, thepreprocessing unit can be configured to register using registrationparameters that are dynamically adjusted using a feedback loop ofcontrol commands to verify, approve and adjust the registrationparameters. In some aspects, the post-processing unit can be configuredto segment the image data and feature extract from the segmented imagedata. In some aspects, the broad analytics unit can be configured tocomparatively process the image data to generate comparative image dataas a portion of the output image data. In some aspects, the device canhave a delta image unit to cross reference and compare segments of theimage data over a time period and across modalities to automaticallymonitor the image data for changes to features of the segments of theimage data. In some aspects, a first segment of the image data at afirst time point can be mapped or masked on subsequent segments of theimage data at subsequent time points. In some aspects, the device canhave a delta image unit to compare a first image segment to a secondimage segment to generate difference image data, the client applicationto generate visual elements of the different image data to visualize alevel of change between the image segments.

Also disclosed herein are processes for transforming image data fordisplay comprising: preprocessing image data by registration andnormalization of the image data; post-processing image data by,transforming the image data to generate output image data, showingchanges in segments of the image data over time; and generating andupdating a graphical user interface of visual elements representative ofthe image output data. In some aspects, a process can comprise comparingfeatures detected using computer algorithms/analytics to subject geneticrisk factors and epigenetics.

Various example details are set forth in the accompanying descriptionbelow. Methods and materials similar or equivalent to those describedherein can be used in the practice or testing of various aspects.Illustrative methods and materials are described by way of example.Other features, objects, and advantages will be apparent from thedescription and from the claims. In the specification and the appendedclaims, the singular forms also include the plural unless the contextclearly dictates otherwise. Unless defined otherwise, all technical andscientific terms used herein have the same meaning as commonlyunderstood by one of ordinary skill in the art.

In some cases, a method described herein can comprise an administrationof a fluorescent dye, for example, ICG, an agent long used forevaluation of the ocular (retinal and choroidal) vasculature, that canbe detected within phagocytic cells such as RPE and macrophages in thehours, days and weeks after systemic vascular administration with theappropriate optical imaging system without further injection of dye.This method is distinct from autofluorescence that relies on endogenousfluorophores without provision of dye, and distinct from angiographythat is used to evaluate the anatomy and patency of the blood vessels,performed during the transit and recirculation phases of dye, and in theminutes or hours immediately thereafter. Such a method, DNIRA, canenhance visualization of the rodent RPE, thereby making patches ofGA-like RPE and photoreceptor loss visible, similar to the method of FAFdescribed for subjects with late dry AMD. Accordingly, some aspects cancomprise a method for enhancing visualization of the RPE/photoreceptorlayer in subjects.

In some cases, a method can comprise visualizing (via generatedgraphical representations), detecting, quantifying the ability ofRPE/retinal cells to internalize/incorporate exogenous dyes as a measureof cellular viability, thus providing for the first time, an image-basedmeasure of cellular activity, metabolism and/or dysfunction. In someaspects, a method can comprise visualizing, detecting, and orquantifying the burden of disease. In some aspects, a method cancomprise detecting tissue at risk of imminent loss. In some aspects, amethod can comprise identifying subjects suitable for clinicalinvestigation or treatment for the secondary prevention of GA amongstthose with early AMD.

In some cases, immune cells, particularly those capable ofinternalizing/incorporating (through, by non-limiting example,ingestion, or passive influx, membrane incorporation) exogenous dyes,fluorophores, chromophores or chemiluminescent compounds, can bedetected in vivo in the eyes of subjects. The data can extend theobservations of ImmunoD and Pulsed DNIRA. Thus, in some aspects, amethod can comprise visualizing, for the first time, phagocytic immunecells in the human eye, including by way of non-limiting example,microglia, macrophages, monocytes, dendritic cells. In some aspects, amethod can comprise co-localization of immune cells with advancingdisease, and thus can suggest their role in disease progression. Suchdisease can include diseases of innate immune dysfunction.

Also disclosed herein are logical algorithms that can distinguish DNIRAfrom other known imaging methods such as FAF, infra-red, or other enfacemethods in any wavelength of light, or cross-sectional methods such asoptical coherence tomography (OCT).

Definitions

The terminology used herein is for the purpose of describing particularcases only and is not intended to be limiting. As used herein, thesingular forms “a”, “an” and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise.Furthermore, to the extent that the terms “including”, “includes”,“having”, “has”, “with”, or variants thereof are used in either thedetailed description and/or the claims, such terms are intended to beinclusive in a manner similar to the term “comprising”.

The term “about” or “approximately” can mean within an acceptable errorrange for the particular value as determined by one of ordinary skill inthe art, which may depend in part on how the value is measured ordetermined, e.g., the limitations of the measurement system. Forexample, “about” can mean plus or minus 10%, per a practice in the art.Alternatively, “about” can mean a range of plus or minus 20%, plus orminus 10%, plus or minus 5%, or plus or minus 1% of a given value.Alternatively, particularly with respect to biological systems orprocesses, the term can mean within an order of magnitude, within5-fold, or within 2-fold, of a value. Where particular values aredescribed in the application and claims, unless otherwise stated theterm “about” meaning within an acceptable error range for the particularvalue should be assumed. Also, where ranges and/or subranges of valuesare provided, the ranges and/or subranges can include the endpoints ofthe ranges and/or subranges.

The terms “subject”, “patient” or “individual” as used herein canencompass a mammal and a non-mammal. A mammal can be any member of theMammalian class, including but not limited to a human, a non-humanprimates such as a chimpanzee, an ape or other monkey species; a farmanimal such as cattle, a horse, a sheep, a goat, a swine; a domesticanimal such as a rabbit, a dog (or a canine), and a cat (or a feline); alaboratory animal including a rodent, such as a rat, a mouse and aguinea pig, and the like. A non-mammal can include a bird, a fish andthe like. In some aspects, a subject can be a mammal. In some aspects, asubject can be a human. In some instances, a human can be an adult. Insome instances, a human can be a child. In some instances, a human canbe age 0-17 years old. In some instances, a human can be age 18-130years old. In some instances, a subject can be a male. In someinstances, a subject can be a female. In some instances, a subject canbe diagnosed with, or can be suspected of having, a condition ordisease. In some instances a disease or condition can be cancer. Asubject can be a patient. A subject can be an individual. In someinstances, a subject, patient or individual can be used interchangeably.

The terms “treat,” “treating”, “treatment,” “ameliorate” or“ameliorating” and other grammatical equivalents as used herein, caninclude alleviating, or abating a disease or condition symptoms,inhibiting a disease or condition, e.g., arresting the development of adisease or condition, relieving a disease or condition, causingregression of a disease or condition, relieving a condition caused bythe disease or condition, or stopping symptoms of a disease orcondition. The term “preventing” can mean preventing additionalsymptoms, ameliorating or preventing the underlying metabolic causes ofsymptoms, and can include prophylaxis.

The terms “Age-Related Macular Degeneration Imaging” (AMDI) and “DelayedNear InfraRed Analysis” (DNIRA) can be used interchangeably to describea method of ocular imaging as described herein.

The term “connected” or “coupled to” may include both direct coupling(in which two elements that are coupled to each other contact eachother) and indirect coupling (in which at least one additional elementis located between the two elements).

FIG. 1 is a flow diagram for processes that are involved in obtaining,identifying, processing DNIRA images, and using them in downstreamapplications. Following image acquisition, cloud-based image analysis isused to generate a complex subject phenotype. Together with genetic andepigenetic data, this subject phenotype may feed the downstream steps ofbiomarker development, improved clinical trial design, drug development,new treatments and ultimately, the selection of specific treatments forindividualized medicine from a menu of options. (QA=quality assurance).

Imaging System

Devices, systems and methods described herein may be implemented in acombination of both hardware and software. Various aspects may beimplemented on programmable computers, each computer including at leastone processor, a data storage system (including volatile memory ornon-volatile memory or other data storage elements or a combinationthereof), and at least one communication interface.

Although exemplary aspects may represent a single combination ofinventive elements, all possible combinations of the disclosed elementscan be envisaged and are within the scope of the disclosure providedherein. For example, if one aspect comprises elements A, B, and C, and asecond aspect comprises elements B and D, then the scope of thedisclosure provided herein can include other remaining combinations ofA, B, C, or D, even if not explicitly disclosed.

Program code can be applied to input data to perform the functionsdescribed herein and to generate output information. The outputinformation is applied to one or more output devices. In some instances,a communication interface may be a network communication interface. Insome instances, elements may be combined. For example, a communicationinterface may be a software communication interface, such as those forinter-process communication. In some instances, there may be acombination of communication interfaces implemented as hardware,software, and combination thereof.

Throughout the discussion, numerous references may be made regardingservers, services, interfaces, portals, platforms, or other systemsformed from computing devices. It should be appreciated that the use ofsuch terms is deemed to represent one or more computing devices havingat least one processor configured to execute software instructionsstored on a computer readable tangible, non-transitory medium. Forexample, a server can include one or more computers operating as a webserver, database server, or other type of computer server in a manner tofulfill described roles, responsibilities, or functions.

The technical solution described herein may be in the form of a softwareproduct. The software product may be stored in a non-volatile ornon-transitory storage medium, which can be a compact disk read-onlymemory (CD-ROM), a USB flash disk, or a removable hard disk. Thesoftware product includes a number of instructions that enable acomputer device (personal computer, server, or network device) toexecute the methods described herein.

Methods described herein can be implemented by physical computerhardware, including computing devices, servers, receivers, transmitters,processors, memory, displays, and networks. The methods described hereincan include useful physical machines and particularly configuredcomputer hardware arrangements for execution of the method. In someaspects, electronic machines and methods implemented by electronicmachines adapted for processing and transforming electromagnetic signalswhich represent various types of information are contemplated.

FIG. 2 is a schematic diagram of a device for processing, storage andretrieval of ocular images according to some aspects. System 100 canconnect to an image acquisition device 106 to receive ocular images ofone or more modalities. System 100 can also connect to an external imagestorage unit 102 to access ocular images of one or more modalities.System 100 can process the images as described herein to generate outputdata. In particular, system 100 can generate graphical representations(of visual elements) of the output data for display on user device 104.The user device 104 can also provide user configurations and parametersto system 100 to provide feedback and input data, as well as dynamicallycontrol the display to update the graphical representations based ontiming and subject data.

System 100 can connect to other components in various ways includingdirectly coupled and indirectly coupled via a network 108. Network 108(or multiple networks) is capable of carrying data. Network 108 caninvolve wired connections, wireless connections, or a combinationthereof. Network 108 may involve different network communicationtechnologies, standards and protocols, such as for example Global Systemfor Mobile Communications (GSM), Code division multiple access (CDMA),wireless local loop, WiMAX, Wi-Fi, Bluetooth, Long Term Evolution (LTE)and so on. Network 108 may involve different physical media such ascoaxial cable, fiber optics, transceiver stations and so on. Examplenetwork types include the Internet, Ethernet, plain old telephoneservice (POTS) line, public switched telephone network (PSTN),integrated services digital network (ISDN), digital subscriber line(DSL), and others, including any combination of these. Network 108 canbe a local area network or wide area network.

FIG. 3 is a schematic diagram of a system 100 for processing, storageand retrieval of ocular images according to some aspects. System 100includes a server 200, database 202, client application 204, userdevices 206, and imaging devices 208. System 100 may be useful for thediagnosis, treatment, or prevention of an ocular disease or disorder.System 100, and in particular imaging devices 208, can use in vivoimaging methods to generate output data. System 100 provides anintegrated platform for computer assisted analysis and diagnosis (CAAD).System 100 receives images of the eye (macula) from various imagingmodalities (captured by imaging devices 208) for comparative processingand provides as output graphical representations of a plurality ofmeasures including prognostic and diagnostic scores upon which to basepersonalized treatments for diseases of the eye. The images can includestandard clinical macular images as well as the novel imaging types asdescribed herein.

System 100 can include a client application 204 to exchange data betweenserver 200 and user devices 206 and imaging devices 108. Server 200 usesas input data images from multiple modalities captured by imagingdevices 108 and user configurations or parameters received by userdevice 206. Server 200 implements image processing to obtainmeasurements and other reporting data for the images. Server 200 has oneor more processing components to register the images, segment objects ofinterest from the images, define patterns of segmented objects and/orintensity texture of the images. The information can be aggregated bythe server 200 into a feature vector for each subject. The subject'sdata can be used as input for statistical models to generate diagnosticand prognostic scores which can be either categorical or continuous innature. System 100 can interact with expert users via user device 206 toprovide training or configuration information for image processing andto the review the outputs from the image processing to provide afeedback loop to check the suitability of the results

Client Application 204 can be either web-based or downloaded and can runlocally on a specified computing machine (user device 206). The userdevice 206 displays a of a graphical user interface (GUI) controlled andgenerated by client application 204. The GUI can display variousgraphical representations of the output data and receive controlcommands to update image processing. Client Application 204 contains aset of processing and analytical tools that provides a platform forimage analysis and storage of ocular images.

The images, once imported, undergo a preprocessing module/stage.Preprocessing may be implemented using a suitable computer system via alocal client application or web-based application. Images may beautomatically normalized and registered. This is a process that isperformed image by image to produce a new set of images (replicates)that have been successfully normalized and registered. Given theinherent variability across images acquired over time, in cases whereimages are not processed well (normalization and/or registrationfailure) the clinician or diagnostician may be prompted or images may betagged to for assessment by the clinical professional at a later date.FIG. 4 is a flow diagram of a method 300 for processing, storage andretrieval of ocular images according to some aspects. At 302, system 100acquires ocular images in various modalities. System 100 can acquireimages using imaging device 208 or from database 202.

In some aspects, system 100 can acquire ocular images in variousmodalities from an external image storage unit 102 or an imageacquisition device 106. At the image acquisition stage, system 100 canacquire images using various ocular imaging equipment. Once acquiredimages are stored locally in a database or on a dedicated local server200 or externally on image storage unit 102. From there clinicians anddiagnosticians can use the user device 104, 206 to import acquiredimages to the client application 204 for further processing andanalysis. The images can be imported by various ways and in differentformats. The images can be sequenced in time for a specific subject seenby a diagnostician (e.g., an ophthalmologist) over a period of time. Theclinician or diagnostician can use user device 104, 206 to import andprocess images as they are acquired on per subject visit basis or importa tranche of images extending over a period of time for multiplesubjects. Image metadata may establish the appropriate sequential timeframe of images and to create a tag for future reference. At 304, system100 can implement preprocessing of the ocular images. The preprocessingmay generally involve registration and normalization of the ocularimages. At 306, system 100 can implement post-processing of the ocularimages. The post-processing may generally involve segmentation andfeature extraction of the ocular images. At 308, system 100 canimplement broad analytics of the ocular images to generate output data.The broad analytics may generally involve comparative processing, and/orreport generation. Server 200 generates graphical representations of theoutput data for display on user device 206. Server 200 stores andretrieves image data and the output data, along with configuration andparameter related data. Database 202 can store historical andcomparative data sets for machine learning and pattern matching byserver 200 in generating the output data. For example, aspects of theworkflow may be implemented by server 200, client application 204, or acombination thereof.

FIG. 5 is a schematic diagram of a system 100 for processing, storageand retrieval of ocular images according to some aspects. System 100 canhave a retinal image capture unit 402, a display device 450, and aserver 200. Server 200 can include various hardware and software units,including, preprocessing unit 404, post-processing unit 408, and a broadanalytics unit 408. Server 200 can store ocular images received fromretinal image capture unit 402. Server 200 can generate graphicalrepresentations of output data for display as part of GUI on displaydevice 450. Server 200 can receive commands, configurations andparameters from display device 450. Server 200 can display processedimages on display device 450. Preprocessing unit 404 generally involvesregistration 410 and normalization 414 of the stored ocular images 412.

Registration 410 can transform different sets of image data into atarget image space, domain or coordinate system, for example. Image datamay be in multiple modalities, data from different image devices, times,depths, or viewpoints. Registration 410 can provide the ability tocompare or integrate the image data obtained from different modalities,formats, and measurements. Registration 410 can receive registrationparameters 422 to transform different sets of image data from the storedocular images 412 from a reference image space to a target image spaceusing transformation models. The transformation models can receive theregistration parameters 422 as input. Feedback can be received toindicate whether or not the registration parameters 422 need improvement420. Preprocessing unit 404 adjusts parameters 416 if control commandsor data processing indicates that they need improvement 420.Registration 410 receives verification or approval 418 before updatingthe stored ocular images 412 with the registered image data.

Normalization 414 can be a process that can change the range of pixelintensity values of the stored ocular images 412. Once registered andnormalized, post-processing unit 460 further processes the storedretinal images 412 to generate unique information regarding the stageand progression of various ocular diseases. The additional processingfeatures can be implemented on a stand-alone basis or sequentially.

Segmentation unit 404 detects and segments regions of the retinal images412. Ocular images inherently possess several characteristic features,such as vessels, that are unlikely to change over time. Ocular imagescan be used to generate visualizations of physiological changesresulting from biological implications. By segmenting areas and featuresof interest on an ocular image the segmented region can be automaticallymonitored over time for changes, such as shape or size. For example, inthe case for a single subject with 4 image acquisitions 1-2 monthsapart, segmentation unit 404 can segment regions of the image andfeatures of interest in the image acquired in the first visit to serveas the baseline for size and shape of feature. When comparativelyanalyzing a sequence of images, the original segmentation can be mappedand/or masked on subsequent images. Additionally, the segmented regionin the first image can be used to quantify changes in size in laterimages.

Feature selection 430 can be used to select and identify features inimages 412 and segments of the images 412. Feature extraction 434 canextract features intended to be informative and non-redundant,facilitating subsequent learning and generalization steps. Featureextraction 430 can be related to dimensionality reduction. Featureselection 430 may facilitate transforming large amounts of image data toa reduced set of features (e.g. a features vector). The selectedfeatures contain the relevant information from the input data, so thatthe comparative processing can be performed by using this reducedrepresentation instead of the complete initial data.

Broad Image Analytics 408 can be used to derive output values based oncomparative processing of images. As images 412 are processed forsubjects on multiple visits, quantitative information 428 can begenerated and stored throughout the process. If a clinician ordiagnostician were interested in compiling and analyzing majorcharacteristic changes over several subjects over time, the data can beselected, pooled and reprocessed to yield aggregate quantitative results428. For example, if a clinician were interested in analyzing changes ingeographic atrophy for a subject pool undergoing a common therapeutictreatment for a specific disease, they can aggregate subject resultsover the course of their visits to yield quantitative measurements andinformation regarding the pool of subjects.

Quantitative metrics 428 can generate different output data values. Anexample quantitative metric 428 is delta. A function of the system 100can be the ability to analyze images, cross reference and comparecharacteristic features across different modalities over time. Featureselection 430 and feature extraction 430 are tools to isolate featuresfrom segments of images 412.

Comparative analyses 446 for feature differences can be performedbetween two images taken sequentially in time within the same modality,but it can also be used to observe and measure differences over imagesseparated by multiple intervening imaging time points (e.g. Time point 1vs Time point 4) and it can be extended to compare images taken fromdifferent modalities, such as fundus auto-fluorescence (FAF), infrared(IR), or DNIRA. Report unit 448 can generate graphical representationsof output data for display on display device 450 as part of GUI. Dataselection 436 selects and isolates data sets for comparison or for useby report generation 448.

FIG. 6 depicts various units and applications of devices and methodsdescribed herein. Exemplary device configuration consistent with FIG. 5are depicted in FIG. 6 and FIG. 7. FIG. 6 is a schematic diagram of asystem for processing, storage and retrieval of ocular images accordingto some aspects. As shown, in some aspects, server 200 may or may notinclude aspects of broad analytics 408. FIG. 6 is a schematic diagram ofa system for processing, storage and retrieval of ocular imagesaccording to some aspects. Raw image input 602 can be provided to animage preprocessing unit 404 to generate qualified image data 608. Imagepost-processing unit 406 further transforms and processes the qualifiedimage data 608 using segmentation 424 and feature extraction 434.

Delta unit 614 can generate delta images as output data. A function ofthe system 100 can be the ability to process images to cross referenceand compare characteristic features of the images across differentmodalities over time. Feature selection 430 and feature extraction 430can be tools to isolate features from segments of images 412.

Comparative analyses for feature differences can be done between twoimages taken sequentially in time within the same modality. Comparativeanalyses can also be used to observe and measure differences over imagesseparated by multiple intervening imaging time points (e.g. Time point 1vs Time point 4) and it can be extend to compare images taken fromdifferent modalities, such as fundus auto-fluorescence (FAF), OCT,fundus photos, red-free, infrared (IR), angiography or DNIRA.

The image differencing or delta process 614 makes use of statistical andalgorithmic processes to compare two, or more than two, preprocessedimages with one another and determine or visualize the level of changebetween them. Differences can also be quantified at a pixel-by-pixel, orregion of pixels, level. In general, two images 412 are compared withone another by subtracting one from the other to generate a differenceimage. This difference image can be an example output data that can beused to generate visual representations for GUI and display device 450.

Delta image generation unit 614 is configured to generate a sequence ofdelta image outputs linked to a comparative time reference, between theimages, either image-by-image or with reference to a common, baselineimage. The selection of images for this delta generation process 614 maybe automated and may also involve manual input. A clinician ordiagnostician can configure the analysis to compare a single, or asequence of images, to a baseline reference and assemble the differencedata in such a way as to highlight major changes over time.

The output Delta images (quantitative outputs 620 and image outputs 618)can be created and stored (either locally or externally). Delta imageoutputs can be viewed and processed through an integrated viewer as partof GUI and displayed on display device 450. When visualized in anintegrated format, the difference images may be overlaid in differentcolors so a user can observe how the changes have manifested themselvesover time. Delta image analysis can also present an overlaid formattingwhere each image is superimposed over the previous one in the visualrepresentation. By allowing for the image differences to be displayed onthe display device relative to the eye structures, the user can beafforded an opportunity to see how the differences compare to thestructure of the eye, making diagnosis of diseases easier.

FIG. 7 is a schematic diagram of a computing device to implement aspectsof system 100 (including server 200 or user device 104, 206) forprocessing, storage and retrieval of ocular images according to someaspects. For simplicity only one computing device is shown but thesystem may include more computing devices operable by users to accessremote network resources and exchange data. The computing devices may bethe same or different types of devices. The computing device maycomprise at least one processor, a data storage device (includingvolatile memory or non-volatile memory or other data storage elements ora combination thereof), and at least one communication interface. Thecomputing device components may be connected in various ways includingdirectly coupled, indirectly coupled via a network, and distributed overa wide geographic area and connected via a network (which may bereferred to as “cloud computing”).

In some aspects, a computing device may be a server, network appliance,set-top box, embedded device, computer expansion module, personalcomputer, laptop, personal data assistant, cellular telephone,smartphone device, imaging device, display terminal, and wirelesshypermedia device or any other computing device capable of beingconfigured to carry out the methods described herein. Computing device(e.g. system 100, server 200 or user device 104, 206) can include atleast one processor 702, memory 704, at least one I/O interface 706, andat least one network interface 708.

Each processor 702 may be, for example, any type of general-purposemicroprocessor or microcontroller, a digital signal processing (DSP)processor, an integrated circuit, a field programmable gate array(FPGA), a reconfigurable processor, or any combination thereof. Memory704 may include a suitable combination of any type of computer memorythat is located either internally or externally such as, for example,random-access memory (RAM), read-only memory (ROM), compact discread-only memory (CD-ROM), electro-optical memory, magneto-opticalmemory, erasable programmable read-only memory (EPROM), andelectrically-erasable programmable read-only memory (EEPROM),Ferroelectric RAM (FRAM) or the like.

Each I/O interface 706 enables computing device to interconnect with oneor more input devices, such as a keyboard, mouse, camera, touch screenand a microphone, or with one or more output devices such as a displayscreen and a speaker. Each network interface 708 enables computingdevice to communicate with other components, to exchange data with othercomponents, to access and connect to network resources, to serveapplications, and perform other computing applications by connecting toa network (or multiple networks) capable of carrying data. A Computingdevice can be operable to register and authenticate users (using alogin, unique identifier, and password for example) prior to providingaccess to applications, a local network, network resources, othernetworks and network security devices. Computing devices may serve oneuser or multiple users.

FIG. 8 depicts an exemplary flow diagram of a method for processing,storage and retrieval of ocular images. The process uses as input afirst image 802 and a second image 804 (and additional images) tocompare features of the images over time. The process generates asoutput visual elements or representations for a GUI to show changes tothe features over the time period.

In some aspects, the process can provide for the comparative analysis ofimages within and between various imaging modalities. The method canprovide for registration 410 two or more images 802, 804 with oneanother by establishing point correspondences between the images or atarget image space. The registration 410 is performed so as to visualizeand address characteristic differences between images taken at aspecific point in time or over a period of time within an imagingmodality, such as individual DNIRA images, or between images ofdifferent modalities, such as DNIRA alongside FAF, OCT, fundus photo,red free, Infrared (IR), angiography, and OCTA images. An example ofregistration is depicted in FIG. 8. The process of registration 410 mapseach point in one image onto the corresponding point in each of theother images. In certain aspects, registration 410 may be accomplishedby determining the registration parameters 422 using correlation and/orfeature location matching. The images may then be registered with eachother via the registration function 410. Alternatively, a mechanicaldeformation model may be used for the registration 410. Differentregistration methods may be employed to register the images with oneanother before comparing the images for differences or changes. Thisincludes fully automatic registration as well as computer assistedmanual registration, or different registration approaches using varyingdegrees of manual intervention. To register more than two images, theimages may be registered in pairs. For example, if a first image and asecond image are registered, and, separately, the second image and athird image are registered, then by the composition of the registrationfunctions, the first and third images are effectively registered. Thisconcept can extend to any number of images. Thus, using a pairwiseregistration 410 process, one can register any number of images.Additionally, registration 410 processes that simultaneously registermore than two images may also be employed by the present technique.

In some instances, registration 410 may be based on landmark extractionor feature-based. The registration 410 of images may be accomplished bymodeling the large-scale motion and local distortion of the anatomy. Insome instances, registration parameters 422 of the model that definesthese motions are estimated and selected. A search is then performed tofind the parameter values that produce the best alignment of the imageswithin and between modalities over time. The quality of this alignmentmay be based on a comparison of corresponding points in the originalimages.

In some aspects of a system described herein, pixel intensities may varysignificantly between images, both within and between imagingmodalities, potentially obscuring valuable characteristics in the image.Normalization 414 of the images can correct for variable intensitiesbetween images. Registration 410 can transform images to a target imagespace for comparison to other images in the common target image space.

Hypofluorescent Signal Processing

Disclosed herein is an en face functional imaging system and method forevaluation of an ocular disease or disorder such as non-exudative (dry)Age Related Macular Degeneration (AMD) capable of identifying bothtissue loss (structural damage) and “sick” or abnormal tissue(functional damage). This method may be referred to herein as AMDImaging (AMDI) or Delayed Near InfraRed Analysis (DNIRA). By way ofillustration the DNIRA method can be compared to Fundus Autofluorescence(FAF). DNIRA, like FAF, can generate images that arereadily-quantifiable using structured computer learning based on thedetection of regions of profound hypofluorescence (black) in otherwisegrey-scale images. FAF may be capable only of quantifying advanced, latedry AMD with Geographic Atrophy (GA) only after irreversible loss of theretinal pigment epithelium (RPE) and photoreceptors.

An imaging system to can be used to register, segment and provideinter-modal comparison between DNIRA and any other imaging methodscapable of generating readily quantifiable images. For example, animaging system can register, segment and provide intra-modal comparisonof DNIRA over time to any other imaging modality over time. In someaspects, an imaging system to register and provide inter- andintra-modal comparison of DNIRA and FAF against any other en face orcross-sectional imaging modalities capable of generating ophthalmicimages. In some aspects, an imaging system to register and provideinter- and intra-modal comparison of DNIRA and FAF against anyfunctional modalities that can generate information about thestate/health of the fundus. By way of non-limiting examples, suchmethods could include microperimetry, low luminance visual acuity, anddark apaptometry.

Imaging system described herein can register and provide inter- andintra-modal comparison of DNIRA, FAF and other imaging modalities,against other presently-used non-imaging modalities such geneticanalysis, not limited by example, comparing to single nucleotidepolymorphisms in genes such as CFH, C2, C3, C5, C6, C7, C8, C9, CFI,CFD, MCP-1, MCP-2, MCP-3, ARMS2/HTRA1, TIMP3, SLC16A8, COL4A3, COL8A1,CETP, LIPC, APOE, VEGFA, VEGFC, ABCA1, ABCA4, MMP9, VTN, NFkappaB.Alternatively images may be compared against genetic analysis of copynumber variation, for example, without limitation, of any of the abovegenes.

Furthermore, imaging systems described herein can register and provideinter- and intra-modal comparison of DNIRA, FAF and other imagingmodalities against other factors, such as analysis of concurrentdisease, for example, but not limiting to, cardiovascular disease, suchas coronary artery disease, atherosclerosis or stroke; neurodegenerativedisease, such as Alzheimer's disease, Parkinson's disease, multiplesclerosis, Huntington's disease, Amyotrophic lateral sclerosis andneurological trauma; inflammatory disease, such as rheumatoid arthritis,osteoarthritis, ankylosing spondylitis, systemic lupus erythematosus,celiac disease, Crohn's disease, Behcet's disease; pulmonary disease,such as asthma, chronic obstructive pulmonary disease (COPD) includingemphysema and allergies; as well as chronic kidney disease. In someaspects, an imaging system can register and provide inter- andintra-modal comparison of DNIRA and FAF against intraocular and systemicbiomarkers, for example, but not limited to, monocyte chemoattractantproteins (MCPs) such as MCP-1, MCP-2, MCP-3, interleukins (ILs) such asIL-6, IL-8, vascular endothelial growth factors (VEGFs), such as VEGF-A,tumor necrosis factors (TNFs) such as TNFa, nitric oxide synthases(NOSs) such as NOS-2, complement components, such as C1-C9, complementregulators, such as CFD, CFH, CFI, apolipoproteins (Apos) such as ApoE,ApoB, C-reactive protein, and eotaxins, such as eotaxin-1. In someexemplary aspects, an imaging system can register and provide inter- andintramodal comparison of DNIRA, FAF and other imaging modalities,against environmental/epigenetic risk factors such as smoking, obesity,blood pressure, body mass index, waist to hip ratio, cholesterol levels,lipid panel, family history, age and dietary vitamin intake andmedications.

Aspects described herein can provide two iteratively-improved dataanalytical processes for testing. A beta version 1 may be configured fornon-cognitive image analysis with iterative improvements based on novelbiology with clinician input; to also include multi-modal imagecomparisons. A beta version 2 may be configured for cognitive andnon-cognitive image analysis with iterative improvements for large dataanalytics, cloud-based analysis; for multi-modal image-basedcomparisons, and for intra-modal comparisons with other subject-centereddatabases (e.g. genetics, concurrent disease, medications, diet).

Aspects described herein can provide an imaging system to implement thefollowing: image output from acquisition device; registration of nativeimages; segmentation of image regions; feature extraction specificdisease-relevant observations (to be compared against other modalities,and over time); inter-modal image analysis and generation of delta imagedata (for example DNIRA subtracting another imaging modality data suchas FAF); and segmentation over time.

When the system 100 generates image output from acquisition device theremay be challenges in the selection of “best” subject images, slowconversion to non-proprietary image output format, long directory names,and a need to be suitable for downstream batch processing. A purpose forgenerating image output from the acquisition device is for selection ofsuitable images for device-independent (agnostic) downstream analysis.The system 100 can be suitable for doctor use, for example, by way of animage-based, interactive GUI interface rendered and updated on userdevice 104, for example.

When the system 100 implements registration of native images there maybe challenges due to the spherical shape of human eye which canintroduces tilt, skew, subtle magnification changes between images, andhuman features potentially introduce significant differences over time(e.g. cataract). A purpose for registration of native images isinter-modal and temporal image comparison. This permits detection ofimage-based features that occur in one or more modalities, and thatincrease, decrease or remain unchanged in size (area) over time. Acomputational requirement is the identification of branch points onretinal blood vessels; visualization of optic nerve head (ONH). Anoptimization may be that images are processed in concentric rings/zonesincluding: a) outer ring discarded due to distortion; b) center ring notused owing to regions of Interest (RoI) and paucity of medium to largeblood vessels. An optimization may be that images are iterativelyreviewed with doctors or experts providing feedback regarding acceptableand unacceptable differences as training input for the machine learningprocesses. An optimization may be for batch processing by way of athreshold value such as of >85% of all images (e.g., to reject <15% formanual registration).

An optimization may be registration of three images, using vascularmarkings (images taken at three separate time points). More or fewerimages may be used in various aspects. FIGS. 17A and 17B show imageoutput data as a visual representation of three images at three separatetime points. The output of registration of native images may be a stackof registered images for downstream manipulation and processing. Thesystem 100 can be suitable for doctor input for iterative improvementsto the parameters. The system 100 can be suitable export-readycloud-based processing for a large data analytical platform.

When the system 100 implements segmentation of the images there may bechallenges after registration to distinguish regions of black fromgrey-scale background. After registration, regions of black shouldexclude constant features such as blood vessels and ONH. A purpose forsegmentation is for identification and quantification of high-contrastfeatures, identified as profoundly hypofluorescent (black). For example,using FAF techniques areas of black correspond to regions ofirreversible tissue loss. FIG. 17A shows progressive, irreversible lossof tissue as visualized by FAF imaging. Over time, areas of blackincrease in size or remain the same (e.g., no regression or reduction inblack is observed).

DNIRA provides a quantifiable, functional readout of tissue function.Using DNIRA, system 100 can detect areas of black in the images thatcorrespond to regions of irreversible tissue loss and to abnormal orsick tissue. Over time, areas of black can remain the same, increase ordecrease in size. System 100 can implement a determination of “baselineDNIRA” images. The system 100 can obtain the baseline DNIRA image bysubtracting signal from pre-dye NIR images from post-dye NIR images.

When the system 100 implements feature extraction there may bechallenges relating to the identification and quantification of featuresof disease other than tissue loss. This process presently relies onmanual interpretation of multi-modal images, including 4 en facemethods, and 1 cross-sectional method. The system 100 can automateaspects of this process using techniques described herein. A purpose offeature extraction is to identify disease-relevant features that assistwith subject diagnosis, staging, risk-stratification, and ultimatelywith clinical trial design and monitoring response to treatment. Apurpose of feature extraction is to make readily possible thequantification and classification of drusen, RPE detachments (RPED), andother disease features using image processing. A purpose of featureextraction is to make possible the observation and quantification ofdynamic changes of disease using graphical representations and visualelements in a GUI displayed on a user device 104. For example, theidentification of the following en face features informs doctors ofdisease severity and subject risk (based on clinical examination andimaging methods): a) small hard drusen; b) medium drusen; c) largedrusen; d) confluent drusen; e) RPE detachments (e.g. drusenoid,serous); f) regions of geographic atrophy (GA); g) blood. System 100implementing DNIRA offers the potential to identify and quantifyclinically-relevant features (beyond just GA) using structuredcomputational processes. This may detect in image data the following: a)small black dots, <65 μm diameter, with circularity of 0.8 or more; b)small black dots, 65-125 μm diameter, with circularity of 0.8 or more;c) large black dots, >125 μm diameter, with circularity of 0.8 or more;d) oblong, oval or round structures deemed to arise from the confluenceof drusen (for example, resembling a shape of “fingerlingpotatoes”), >125 μm in one dimension, with circularity >0.2; e) largeblack areas >200 μm in diameter, circularity >0.5; f) regions of black,seen in both DNIRA and FAF; g) blood.

The system 100 implements inter-modal image analysis and generates deltaimage data (DNIRA subtracting FAF). The delta image, defines, at anyimaging session the regions of black DNIRA that correspond with black GA(and so represent GA), versus the black DNIRA that is in excess of theblack FAF (so represents potential damaged or abnormal tissue, presentoutside of GA), for example. En face segmentation-suitable methodsinclude DNIRA and FAF. The system 100 implements inter-modal imageanalysis and generates delta image data to identify disease-relevantfeatures in registered and segmented images, both between modalities(FAF & DNIRA only) and over time. The system 100 implements inter-modalimage analysis and generates delta image data for the identification ofen face features informs doctors of disease severity and subject riskbased on FAF, including for example regions of geographic atrophy (GA).DNIRA offers the potential to identify and quantify GA using structuredcomputational processes. For example regions >200 μm, or alternativelymeasured relative to the diameter of the ONH, that appear black usingDNIRA and are equal or larger in area to black regions also observed onFAF in the same spatial location. The output can be Delta Image datawhich is “DNIRA minus FAF” image data to identify regions ofcorresponding black signal.

The identification of en face features is not presently known to doctorsand may inform them of abnormal RPE/PhR physiology in regions outside ofGA (e.g., the extent of diseased tissue, not dead tissue), including forexample regions of abnormal cellular physiology that do NOT include GA.Discrete regions >200 μm (or measured relative to the ONH) that appearblack using DNIRA may not be evident on FAF which provides animprovement. Delta Image data includes “DNIRA minus FAF” image data toidentify regions of black signal that extend beyond the borders of GA,or exist in different spatial areas.

En face grey-scale methods include: InfraRed (IR) Reflectance, Red Free(RF) Color, Color (RGB), Color (CNYK), angiography [useful for wet AMD;but may be less useful for dry AMD], en face Optical CoherenceTomography (OCT) and OCT angiography (OCTA). A purpose of inter-modalimage analysis is to identify disease-relevant features in registeredimages, both between modalities and over time, in grey-scale images. Theinter-modal image analysis may implement the identification of thefollowing en face features informs MDs of disease severity and subjectrisk on examination and in currently available en face imaging methods(excluding FAF): a) small hard drusen; b) medium drusen; c) largedrusen; d) confluent drusen; e) RPE detachments (drusenoid, serous,hemorrhagic); f) blood; g) regions of geographic atrophy (GA).inter-modal image analysis can use unstructured computational processes(e.g. cognitive computing), to generate output data and transform imagedata.

Cross-sectional methods include: Optical Coherence Tomography (OCT) andOCT angiography (OCTA). The inter-modal image analysis may implement theidentification of the following cross-sectional features, detected onOCT, informs MDs of disease severity and subject risk: a) presence andnumber of drusen in RPE/sub-RPE space; b) drusenoid RPE detachments; c)pseudodrusen in subretinal space; d) geographic atrophy (loss of RPE/PhRwith greater OCT signal in deeper tissues), e) drusen volume, f) areasof drusen regression, g) outer nuclear layer (ONL) thickness and volume,h) hyper-reflective OCT foci, i) presence of nascent GA, j) presence ofquiescent CNV k) sub-foveal choroidal thickness, l) collapsed pigmentepithelial detachment En face DNIRA or other modality imaging can becompared against OCT using single cross-sections or 3D reconstructionsthat generate graphical representations of the cross-sectional imaging.

The system 100 can implement segmentation over time where differentfeature that are selected and extracted are followed over time (e.g. todetermine if it is increasing, remaining the same, or deceasing in areaor number). The system 100 can implement segmentation and, followed overtime, FAF can enable doctors to quantify regions of GA and measure theirrates of expansion over time. This step can permit identification ofsubjects with highest rates of GA expansion, and may be useful forclinical trial enrollment and endpoint analysis. However it may notfollow changes before irreversible tissue loss. Further, thequantification of GA suggests that disease is unidirectional (e.g. thatfeatures accumulate but do not regress) which may be false. Three mainparameters can be assessed: A. change in total black DNIRA (beyond GA);B. change in total drusen; C. change in RPE detachments. A purpose ofsegmentation over time may be to quantify clinically-relevant feature ofdisease (e.g. drusen, RPEDs, etc.) and determine their change over time.including Areas of GA; Total area beyond GA (as potential measure ofdisease burden); and Areas of subsegmented features of disease (e.g.small, med, large drusen, RPEDs, etc). Data confirms that regions ofblack DNIRA not only increase, but can decrease over time, confirmingthe dynamic features of disease observed most typically in color photos(for example drusen regression). This may be drusen regression. Thesystem 100 implements segmentation over time to generate Delta Time(DNIRA) which can be measured as a decrease of black DNIRA signal overtime in small, medium or large. The system 100 implements segmentationover time to generate the following output a) Changes in total blackDNIRA over time; b) Changes in “Delta DNIRA minus FAF” over time; c)Changes in specifically segmented features, for example: changes insmall drusen, changes in medium drusen, and changes in large drusen.

The change in disease burden (outside of GA) may not be detected usingknown approached. To identify total diseased or dysfunctional tissuethis parameter could provide a measure of the burden of disease andsupport clinical trial enrollment. The output of “Delta Image=DNIRAminus FAF” can indicate regions that increase in size (area) over time.

The system 100 implements segmentation over time to quantify the extentof diseased or dysfunctional tissue. The output “Delta Image=DNIRA minusFAF” indicate regions that remain unchanged in size (area) over time.The output “Delta Inage=DNIRA minus FAF” indicate regions that decreasein size (area) over time.

The change in drusen accumulation over time may be difficult to identifyand even more difficult to quantify. Drusen are not uniformly detectedon FAF or other en face methods. DNIRA may be used to detect andquantify dynamic changes in drusen accumulation & regression. It iswidely held that the accumulation of drusen (number, area or 3D volume)is proportionate to disease. The output “Delta Image=DNIRA minus FAF”indicate regions that increase in size (area) over time. The output“Delta Image=DNIRA minus FAF” indicate regions that remain unchanged insize (area) over time.

Drusen regression (often seen in color photos or by number, area or 3Dvolume) can be a sign of progressing disease. Drusen loss can heraldlate AMD, with GA or CNVM. The output “Delta Inage=DNIRA minus FAF”indicate regions that decrease in size (area) over time.

Drusen may be detected using OCT, with 3D slab analysis. It is held thatthe accumulation of drusen (number, area or 3D volume) can beproportionate to disease. The output “Delta DNIRA minus FAF” indicateregions that increase in size (area) over time which may be used forcomparing against OCT.

The change in RPE detachment (RPED) size (area, height, volume) overtime may be difficult to identify and difficult to quantify. The system100 can detect and quantify dynamic changes in RPED size (area) andnumber. It is held that the resolution of RPEDs, [particularly thedrusenoid variety] suggests risk for progression to late, blinding GA.The output “Delta Image=DNIRA minus FAF” indicate regions of RPED thatincrease in size (area) over time. The output “Delta Image=DNIRA minusFAF” indicate regions that remain unchanged in size (area) over time.The output “Delta Inage=DNIRA minus FAF” indicate regions that decreasein size (area) over time.

Although the accumulation of drusen reflects ongoing disease, regions ofdrusen regression (often seen in color photos, or by number, area or 3Dvolume) is a sign of progressing disease. Drusen loss can herald lateAMD, with GA or CNVM. In some aspects, AMDI can detect regression ofdrusen, allowing determination of progression of AMD.

In some aspects, AMDI images can be compared against other imagingmethods such as OCT-Angiography (OCT-A). Such a comparison allowsidentification of regions of pathological blood flow (either decreasedflow through the choriocapillaris or other layers of the choroid) orpotentially increased flow into areas of leaking or non-leakingneovascularization (choroidal neovascularization) or vascular pathology(such as choroidal plexus, polyps, aneurysms, anastomose).

Exemplary Output Data

FIGS. 11A-11D depict exemplary demonstrations of visual representationsthat can be visualized using a system described herein. FIG. 11A showsimage output data as a visual representation of multiple small andmedium black dots (<65 μm and 65-125 μm). FIG. 11B shows image outputdata as a visual representation of a large drusen (>125 μm). FIG. 11Cshows image output data as a visual representation of confluent drusen.FIG. 11D shows image output data as a visual representation of GA.

In some cases, output data can be depicted as multiple black dots asrepresented in FIGS. 12 and 13. Unlike FAF, the DNIRA signal is blackwhere RPE is separated from BM & choroid by structures such as drusen(arrows overlaid on the DNIRA image show representative drusen). Asdepicted in FIG. 13, soft confluent drusen demonstrate oblong footprintsthat are profoundly hypofluorescent (black) using DNIRA. Soft drusen aretypically defined by colour fundus photography. They are grey-scale, andhence not quantifiable in other wavelengths.

In some cases, output image data obtained by DNIRA can be comparedoutput image data obtained by other methods to elucidate features of theimage pair that would not necessarily be apparent when analyzing theimages in isolation. For example, FIG. 12 depicts comparative retinalimages obtained by IR, FAF, DNIRA, and OCT imaging. Arrows indicate thelocation of drusen visible in the DNIRA image. These drusen are notreadily visible in the corresponding IR or FAF images. FIG. 14 alsoshows differences between images acquired using DNIRA with imagesacquired by FAF. FIG. 14 depicts a representative DNIRA image of apatient with central GA observed on FAF (left). As depicted in FIG. 14,image output data from FAF imaging (left) shows less black (GA) while aDNIRA image (right) shows more black. The DNIRA image (right) showsgreater number of hypofluorescent signals, a greater total area ofhypofluorescence, and greater amount of perimeter/border ofhypofluroescence. The difference between these two images represents the“Delta” portion of the signal. The Delta Image data can be “Area ofDNIRA black—(minus) Area of FAF black” which is the area of diseased ordysfunctional tissue, in absence of GA.

FIG. 15A shows image output data as a visual representation with anupper image representing a large RPE detachment appears black usingDNIRA and a lower image representing OCT shows elevation of RPE layer(elevated thick white line). Blue horizontal line indicates level atwhich OCT cross-section was obtained. FIG. 15B shows image output dataas a visual representation with a left image having an upper portion forDNIRA that shows region of black and a lower portion for OCT that showssmall drusenoid RPE detachment (definition >200 μm). A right image hasan upper for DNIRA that shows some remaining black signal and a lowerportion for OCT that shows resolution of RPE detachment. Note that theright image also shows association of bright dots with RPED suggestingthe presence of macrophages.

In some aspects, analysis of image data from multiple imaging methodscan be used to visualize features of an RPE. FIG. 16 shows image outputdata as a visual representation with the left image for IR, the middleimage for FAF, and the right image for DNIRA. Hypofluorescent (black)regions indicate loss of RPE.

In some instances, images can be collected as a function of time inorder to elucidate changes in a disease state or condition. For example,FIG. 17A shows image output data as a visual representation of a leftimage a time t=1, a middle image at t=2, and a right image at time t=3(all 4 months apart). The region of black is greater in first image thanthe second, indicating that feature is decreasing. FIGS. 18 and 19A-19Jshow assembly of TAMi composites can allow for correlation of thedisclosed imaging method with current clinical modalities. Fundusautofluorescence (FAF, FIG. 19B) and TAMi composites (FIG. 19C) wereassembled as an overlay on color photographs (FIG. 19A). Multi-modalanalysis allowed identification of regions of interest (FIG. 19F)including the tumor region (i), areas of current or previous fluid (ii),and peripheral (extra-lesional) regions (iii). Region area in mm² wasdetermined using ImageJ measurements and used in hyperfluorescent dotdensity calculations (FIG. 19H). These were compared across patientswith melanomas, indeterminate lesions, or benign nevi, as determined byclinician ground truth. Mean dot density is shown as number ofdots/mm2±SEM. One-way ANOVA and post hoc multiple comparisons (FIG. 19I)show significantly higher dot densities in melanoma arms whenconsidering lesion and fluid regions, but not in peripheral regions.(FIG. 19J) Multiple comparisons of dot density in each region withinrisk group's fond melanoma had significantly higher dot density inlesion and fluid regions when compared to peripheral areas. This was notobserved in other risk groups.

Hypofluorescent DNIRA Signal

In some aspects, a method can comprise identifying discrete regions ofthe eye, for example within the macula, where a dye, fluorophore,chromophore or chemiluminescent molecule that should otherwise beinternalized, taken up, processed or accumulated, cannot be, (or can toa lesser extent) thus precluding or reducing its detection. This mayoccur within particular layers or tissues such as, by way ofnon-limiting example, discrete regions of the RPE monolayer. Suchregions, made visible by their hypofluorescent/black signal, can beidentified and quantified by DNIRA/AMDI, both within a single image andtheir rate of change calculated across multiple images over time.

Similar observations have been made using for example FAF, in whichregions of absent signal correspond spatially with regions ofRPE/photoreceptor loss, as occurs with Geographic Atrophy. FAF detectsan endogenous fluorophore produced by the photoreceptors and accumulatedwithin RPE, and so too when RPE is lost, the FAF signal is lost. Becauseregions of GA cannot regress, e.g., become smaller in size, area ofprofoundly hypofluorescent/black FAF either remain the same or increasein size over time.

Aspects described herein relate in part to the temporal changes in DNIRAimages that demonstrate expansion over time. Without wishing to belimited by theory, such increases in the areas of hypofluorescent/blackDNIRA could represent increased burden of disease. As such these regionscould represent areas where abnormal or damaged RPE/photoreceptors arenot able to internalize dye. Again, without wishing to be limited bytheory, such increases could (i) represent regions where changes intissue anatomy preclude or reduce dye uptake in the presence ofotherwise normal RPE/photoreceptor cells or (ii) represent regions wherethe anatomy is normal, but the cellular physiology is abnormal.

In some aspects, temporal changes in DNIRA images can be employed todemonstrate reduction of profoundly hypofluorescent/black areas overtime. Without wishing to be limited by theory, such decreases in theareas of hypofluorescent/black DNIRA could represent decreased,improved, or lessened burden of disease. As such these regions couldrepresent areas where abnormal or damaged RPE/photoreceptors are betterable to internalize dye. Again, without wishing to be limited by theory,such reductions in hypofluorescent/black signal could also representregions where changes in tissue anatomy are improved, therebyfacilitating uptake of dye in the presence of otherwise normalRPE/photoreceptor cells. As example, such reduction could occur withresolution of an RPE detachment (RPED) that permits the RPE layer tore-appose itself with its underlying Bruch's membrane, and so facilitatetransfer and uptake of dyes such as ICG from the underlying choroidalvasculature. Such reduction in the size of hyperfluorescent/blackregions is not reported using FAF or other imaging modalities. As such,these features demonstrate the dynamic quality of DNIRA, distinguishingit further from FAF and other static imaging modalities.

Aspects described herein can relate to temporal changes in DNIRA/AMDIthat demonstrate the persistence of profoundly hypofluorescent/blackregions that remain unchanged over time. When coincident with profoundlyhypofluorescent/black regions of FAF, they are hypothesized to reflectregions of RPE/photoreceptor loss, after it is dead and gone. Bycontrast, other regions of profoundly hypofluorescent/black regions ofDNIRA may reflect, without wishing to be bound by theory, anatomicalareas that preclude the uptake of dyes such as ICG from the underlyingchoroidal vasculature and that they remain largely unchanged. Forexample, in addition to RPE detachments that separate theRPE/photoreceptors from the underlying choroidal vasculature, theinterposition of small deposits known as drusen, basal laminar depositsor basal linear deposits, could also preclude uptake of dye (FIG. 10).As such, the presence of profoundly hypofluorescent/black dots, measuredto be consistent in size and location with drusen observed in otherimaging modalities (e.g. IR, red free, color fundus photography, OCT,angiography, OCTA), could provide a novel method for measuring totaldrusen area. Without wishing to be bound by theory, drusen volume isconsidered an overall measure of the burden of disease, but theirquantification is difficult owing to their grey-scale nature usingimaging methods such as IR or FAF, and the need for 3-dimensional (3D)volumetric reconstruction in other methods such as OCT. In this aspect,the ability to quantify small, profoundly hypofluorescent/black dots ina given size range, for example larger than a droplet (a smallnon-pathological aging deposit) and smaller than a drusenoid RPED (e.g.200 to 250 μm) could provide an entirely novel method of quantifying theburden of disease.

Mid-Grey Hypofluorescent Signal

In some instances, AMDI identifies regions of mid-tone grey—neitherprofoundly hyperfluorescent/black nor brightly hyperfluorescent. These2D regions occur in some but not all subjects, and may occur in thepresence or absence of pre-existent GA (as determined by FAF forexample). Unlike the profoundly hypofluorescent/black regions, nostructural abnormality (e.g. no drusen, no RPE detachments, no GA) wasobserved. Instead, RPE cells and the retina appear anatomically normalboth using other en face modalities (e.g. IR, FAF, red-free, color,angiography, OCTA and others) and in cross-section (e.g. OCT). Thesecells may be physiologically deranged, damaged, or genetically distinctfrom normal cells, as so are unable to internalize dyes such as ICG. Assuch, in some aspects, AMDI can be used to identify and quantifydysfunctional RPE/retina.

FIG. 20 shows comparative retinal images obtained by IR, FAF, and DNIRAimaging. Unlike FAF, regions of apparently “normal” RPE appear anintermediate grey in the DNIRA image. Unlike FAF, the DNIRA imagereveals abnormal or “sick” RPE as a darker grey (highlighted by theyellow line) against an intermediate grey background.

Hyperfluorescent Signal Processing

In some aspects described herein, an in vivo functional imaging methodcan be employed for the detection of phagocytic immune cells, e.g.macrophages or microglia (collectively macrophages), in the eyes ofsubjects with an ocular disease or disorder. The system 100 implementingDNIRA identifies presumptive macrophages in the eyes of subjects withintra-ocular tumors and AMD, placing these potentially aggressive cellsin proximity to regions of tissue damage and inflammation. Further,complex 2D patterns of hyper-, normo- and hypo-fluorescent areidentified for the first time using DNIRA—these grey-scale images may beused by system 100 to implement cognitive computing for their analysisand interpretation.

At present, there may be no method for detecting immune cells in theliving human eye, and because tissue biopsy is not a viable option, nomethod for providing clinico-pathological correlation between in vivofindings and post-mortem histopathology exists.

In some cases, system 100 can implement the following: 1) usingregistered DNIRA images, to identify the spatiotemporal distribution ofpresumptive macrophages; 2) using registered DNIRA images, to identifythe co-distribution of presumptive macrophages with high-risk andlate-stage features of disease; 3) using registered DNIRA images, toidentify the change in the spatiotemporal distribution andco-distribution of macrophages and high-risk and late-stage features ofdisease over time; 4) to identify novel complex patterns of diseasephenotype; and 5) to compare DNIRA images against other en face andcross-sectional modalities over time. System 100 can generally implementimage output processing from the image acquisition unit; registration ofnative images; and segmentation of images. System 100 can generallyimplement image output processing from the image acquisition unit. Theremay be challenges relating to a) selection of “best” subject images; b)slow conversion to non-proprietary image output format; c) longdirectory names; and d) need to be suitable for downstream batchprocessing. System 100 can generally implement image output processingfor selection of suitable images for device-independent (agnostic)downstream analysis and processing. System 100 can generally implementregistration of native images.

System 100 can generally implement segmentation of images. There may bechallenges as after registration. Individual bright dots may bedistinguished from areas of surrounding black and grey-scale background.Further bright presumptive macrophage dots may be quantified over time.System 100 can generally implement segmentation of image to identifyspatiotemporal distribution of bright macrophages. Macrophages arebelieved to be associated with disease activity. System 100 cangenerally implement segmentation of image for selecting regions ofinterest “ROIs”. System 100 can generally implement segmentation ofimage for thresholding bright signal to separate from otherwisegreyscale background. System 100 can generally implement segmentation ofimage for identifying appropriate dot size. System 100 can generallyimplement segmentation of image for quantifying numbers of bright DNIRAmacrophages in ROIs. System 100 can generally implement segmentation ofimage to -indicate range of allowable error for thresholding to capturebright signal. System 100 can generally implement segmentation of imageto indicate range of allowable error for capturing dot size.

System 100 can generally implement segmentation of image to correlatedistribution of bright macrophages with regions of high-risk diseasefeatures. High risk features of disease can predict disease progression;therefore correlating macrophages to these high risk features mayprovide another biological predictor of disease and increase thepredictive power. System 100 can generally implement segmentation ofimage for selecting ROIs in areas of disease activity. System 100 cangenerally implement segmentation of image for quantifying numbers ofbright DNIRA macrophages in ROIs using processes described herein.

System 100 can generally implement segmentation of images to identifythe change in the spatiotemporal distribution and co-distribution ofmacrophages and high-risk and late-stage features of disease over time.Macrophages are highly plastic and dynamic cells that change spatiallyand temporally based on their tissue environment, therefore theirchanges can further help predict future disease activity. System 100 cangenerally implement segmentation of images for thresholding brightsignal to separate from otherwise greyscale background. System 100 cangenerally implement segmentation of images for identifying appropriatedot size. System 100 can generally implement segmentation of image forcapturing bright dots in ROI to quantify. System 100 can generallyimplement segmentation of images for comparing bright dots in ROI fromone timepoint to the next which may indicate a change in quantity(increase/decrease over time), or a change in localization within thesame ROI (no increase/decrease but spatial change).

System 100 can generally implement segmentation of images to identifycomplex patterns of disease phenotype. Imaging macrophages in vivo inthe subject eye has never been reported before, and disclosed herein aremethods that identify new patterns that have not been previouslyidentified and need to be characterized. System 100 can generallyimplement segmentation of images for thresholding bright signal toseparate from otherwise greyscale background. System 100 can generallyimplement segmentation of image for identifying appropriate dot size.System 100 can generally implement segmentation of image for identifyingclusters of grouped dots. System 100 can generally implementsegmentation of image for identifying commonality in clusters of dots todepict distinct patterns that share features across subjects. Forexample, a similar numbers of grouped dots, similar distances betweenclusters, and similar patterns of localization may provide anindication.

System 100 can generally implement segmentation of image to compareDNIRA images against other en face and cross-sectional modalities overtime. DNIRA provides features that are unprecedented by other ophthalmicmodalities, and by comparing to other modalities can increase the powerassociated with DNIRA, but also help identify potential features inthose modalities, the biology of which has been unknown before. Examplecomparisons include DNIRA vs FAF, DNIRA vs OCT, DNIRA vs IR, DNIRA vsNIR-AF, DNIRA vs RF, DNIRA vs angiography, DNIRA vs OCTA.

The following provides an example of registration and analysis withreference to FIG. 20 which provides an example graphical representationof image data. The system 100 can register each timepoint using thelinear stack alignment with SIFT plugin with the following (mostlydefault) settings (determined through trial-and-error): Initial Gaussianblur: 1.60 pixels, Steps per scale octave: 3 pixels, Feature descriptorsize: 4 pixels, Feature descriptor orientation bins: 8, Closest/nextclosest ratio: 0.92, Maximal alignment error: 25 pixels, Inlierratio=0.05. The expected transformation may be Affine. The system 100can subtract a blurred version of each image from the original image.This should remove most of the image background, but the contrast may below as a result. A blurred image provides a Gaussian blur with sigma=50μM. The system 100 can subtract remaining background using “SubtractBackground” plugin with rolling ball radius=4 μM. This helps remove lastvestiges of background. The system 100 can enhance contrast, saturating0.2% of pixels, and normalize the histogram. The system 100 can stretchhistogram of timepoint 2 to mimic pixel intensity of timepoint 1 usingthe “Bleach Correction” plugin. This is done because differences inaveraging number, shadows, and more can alter relative brightness ofdots between two timepoints. Since a threshold may be applied, dots maybe similarly bright in both images. This plugin was originally intendedto compensate for diminishing signal intensity when imaging sensitivefluorophores over long periods of time. The system can apply“Intermodes” thresholding algorithm, and convert to mask. In this step,anything above a threshold intensity value as determined in the“intermodes” algorithm can be captured, and everything else can beremoved. In theory, this should select only the brightest signals. Insome aspects, this selects all of the dots.

Furthermore, system 100 can despeckle, watershed and dilate (despeckleto remove noise, watershed algorithm to separate close-together dots,prevents the program from wrongly identifying a cluster of dots as asingle large dot). The system 100 can use “image calculator” feature togo pixel-by-pixel, selecting for “max” and “min” brightness valuesbetween two timepoints with an explanation for this is outlined in thenext slide. The system 100 can subtract a “max” image from a “min”image. System 100 can analyze particles with following settings Size<400 μM², circularity=0.50-1.00. Back-calculating, 400 μM² leads to adiameter of 22 μM. This is reaching macrophage territory in terms ofsize, but this also reaches the current resolution limit of DNIRA. Theblur associated with the point-spread function means that the dots inDNIRA are likely larger than the source of said dots. System 100 can useappropriate means to conduct a Manders co-localization test. This is aco-localization test commonly used for IHC, but is now being applied totest the system's ability to identify static vs dynamic dots.

Hyperfluorescent DNIRA Signal

In some aspects, DNIRA can identify small hyperfluorescent dots in theeyes of subjects. Surprisingly, some of these dots are stationarybetween subject visits, while others are dynamic, being present at sometimepoints and not at others, residing at different locations. Withoutwishing to be bound by theory, these small hyperfluorescent dotsidentified using DNIRA may represent phagocytic immune cells, such asmacrophages, that migrate, divide, appear or disappear over time. It waspreviously demonstrated preclinically using DNIRA and two variants ofDNIRA—ImmunoDNIRA and Pulsed-DNIRA—that cells such as macrophages caninternalize ICG and be visualized thereafter in the living eye usingexcitation/emission filters coincident with the spectral characteristicsof the dye. Disclosed herein is the unprecedented finding that thestationary or dynamic population of dots identified in the eyes ofsubjects with AMD are of appropriate size and shape to be immune cellssuch as macrophages.

With reference to FIGS. 18, and 19, by way of proof-of-concept, it isalso observed that similarly-size hyperfluorescent dots are present ineyes with uveal melanoma or high-risk indeterminate melanocytic lesions.They are absent from nevi and low-risk indeterminate lesions.Inflammation is a hallmark of cancer, and pathological specimens ofuveal melanoma, particularly those of high-grade (chromosome 3 monosomy)are associated with high numbers of macrophages while nevi are not. Thiscorrelation is confirmed in vivo using DNIRA to visualize TumorAssociated Macrophages (TAMs) by way of the method termed “TAMI” (TAMImaging). Using TAMI it is confirmed that regions of interest can besegmented based on their presence or absence of hyperfluorescent DNIRAlabelled dots. These regions correspond to differences in theconcentrations of bright dots, which can be quantified. Hyperfluorescentdots may be identified by number, density, concentration, size orradius, fluorescent intensity, or location. The quantification of dotscorrelates significantly with burden of disease, as regions with largeramounts of dots correlate to incidence of melanoma, compared toindeterminate lesions or benign nevi.

2D Patterns of DNIRA Signal

In some aspects, early studies described herein applying DNIRA/AMDI tosubjects have identified a novel two-dimensional (2D) grey-scalepattern, as observed in FIGS. 21A-21E. These can occur centrally withinthe macula, or more peripherally in the macula or the mid-retina. Thesepatterns often have an interwoven, lacy, reticular or spot-likeconfiguration. In some instances are observed patterns that are morecoarse (wider) in aspect or finer (tighter) in aspect. Without anyprecedent, in some cases these are termed “loose weave” (FIG. 21A) or“tight weave” (FIG. 21B), respectively. These patterns may be indicativeof different subtypes of AMD and therefore different response totherapeutic options, as depicted in Example 7.

It therefore follows, that in some aspects, logical and softwarealgorithms may be used to identify and categorize or stratify subjectsby their 2D patterns observed using DNIRA/AMDI. As these patterns arenot readily described by conventional software programming (e.g.registration and segmentation), unstructured computational algorithmsare applied to refine their description.

Delta—Comparing FAF and DNIRA

In some aspects, there is provided a method for distinguishing regionsof profoundly hypofluorescent/black signal in FAF (whereRPE/photoreceptors are dead and gone) from regions that are profoundlyhyperfluorescent/black in DNIRA. Compared against FAF, DNIRA imagesalways have the same or greater amounts of profoundlyhypofluorescent/black areas than does FAF, enabling us to calculate thedifference, or delta, as depicted in FIG. 14. This delta may representRPE/photoreceptors with abnormal physiology rather than their loss.Thus, in some aspects, DNIRA can detect regions of disease.

Some aspects can relate to the changes in DNIRA images compared againstother imaging modalities (for example, FAF, IR, color, red free,segmented or non-segmented cross-sectional or en face OCT, OCTAproviding the difference between the two modalities, delta).

It additionally relates to the temporal changes of DNIRA over timecompared against the temporal changes of FAF over time. For example, andwithout wishing to be limited by theory, regions of profoundlyhypofluorescent/black DNIRA seen at timepoint 1 may precede, and bepredictive of changes seen in other modalities such as FAF at a latertime. As such, DNIRA can be seen as a method for detecting early change.It therefore follows, that in some aspects, DNIRA can identify regionsof RPE/photoreceptor damage before its loss and before the onset ofprofoundly hypofluorescent/black regions of RPE/photoreceptor lossdetected using FAF. Accordingly, DNIRA/AMDI can therefore be used toidentify subjects at risk of developing GA, e.g., progressing from earlyto late dry AMD. As such, DNIRA provides a novel method of identifyingsubjects for enrollment in clinical trials to prevent the onset of GA,or for treatment should one arise.

Logical Biology-Driven Hardware and Software for Image Processing

Aspects described herein relate to the logical algorithms used to drivesoftware and hardware design. These include, but are not limited tothose described above, e.g. temporal changes in DNIRA and cross-modalitydifference between DNIRA and other methods, and the temporally-relatedcross-modality changes. AMDI has not previously been reported. Exemplarydata disclosed herein are derived from the first application of DNIRA tothe clinical setting. Accordingly, each of these observations is novel.In some cases, systems described herein can comprise software modulesthat implement the logical processes described herein. Furthermore, asystem described herein can comprise an integrated image processing,image analysis and image output platform.

In some aspects, such logical and software algorithms can identify,quantify and align regions of profoundly hypofluorescent/black DNIRAover time and between modalities. In some aspects, such logical andsoftware algorithms can identify, quantify and align hyperfluorescentdots, evaluating their numbers, size and location. Such logical andsoftware algorithms can identify, quantify and align brighthyperfluorescent dots relative to regions of profoundlyhypofluorescent/black DNIRA over time and between modalities. Withoutwishing to be bound by theory, spatiotemporal localization ofpresumptive immune cells such as macrophages in proximity to areas ofRPE/photoreceptor loss, implicates phagocytic activity withRPE/photoreceptor loss. Co-localization is described herein. As such,DNIRA could be used to identify subjects suited to immunomodulatorytherapies.

In some aspects, the ocular disease or disorder can be one or more ofdry AMD, RPD, white-dot syndromes (e.g. serpiginous chorioretinopathy,serpiginous retinopathy, acute posterior multifocal placoid pigmentepitheliopathy (APMPPE), multiple evanescent white dot syndrome (MEWDS),acute zonal occult outer retinopathy (AZOOR), punctate innerchoroidopathy (PIC), diffuse subretinal fibrosis (DSF)), late onsetretinal degeneration (LORDs; e.g. Q1qTNF5 deficiency), and centralserous retinopathy (CSR). In some aspects, the ocular disease ordisorder can be Lecithin Retinol Acyltransferase (LRAT) deficiency,which is optionally associated with: lrat-related leber congenitalamaurosis, and retinal dystrophy, early-onset, severe. In some aspects,the ocular disease or disorder can be fundus albipunctatus, which may beassociated with one or more of the following genetic locations: 3q22.1(Retinitis punctata albescens, RHO); 6p21.1 (Retinitis punctataalbescens, PRPH2); 12q13.2 (Fundus albipunctatus, RDH5); 15q26.1(Retinitis punctata albescens, RLBP1); and 15q26.1 (Fundusalbipunctatus, RLBP1). In some aspects, the ocular disease or disordercan be one or more of dry AMD and RPD disease. In some aspects, thepresence of phagocytic immune cells is measured by DNIRA.

In some aspects, the ocular disease or disorder can be one or more of adiabetic eye disease (for instance, diabetic retinopathy and DME),Vogt-Kayanagi-Harada disease (VKH); Sarcoid uveitis; Ocularhistoplasmosis and/or Presumed Ocular Histoplasmosis Syndrome;idiopathic uveitis, Autoimmune uveitis; Uveitis associated with systemicdiseases, e.g. lupus, Crohn's disease, rheumatoid arthritis, and otherdiseases of known immune origin; Posterior uveitis (including that whichmay not yet be diagnosed); Anterior uveitis (e.g. iritis); Bechet'sdisease; Polyarteritis nodosa; and Wegener granulomatosis. This isdepicted in Example 21, and FIG. 39.

In some aspects, the RPD disease can be identifiable by the presence ofone or more areas of distinct patterns of retinal imaging in the eye ofa subject. The retinal imaging can be one or more of white light,red-free light, blue light, FAF, near infra-red (NIR), infra-red (IR),angiography, and DNIRA and/or the presence of one or more areas ofabnormally-fluorescent FAF in the eye of a subject and/or an increase(including a transient increase) in permeability across the subject'sepithelial barrier between a choroid and a retina relative to anundiseased state and/or a presence of phagocytic immune cells across thesubject's RPE relative to an undiseased state.

In some aspects, a method can determine whether an ocular disease ordisorder in a subject is responsive to treatment with an agent thatinhibits or modifies the function of a subject's immune cells,comprising detecting a presence, detecting an absence, or measuring anamount of immune cells in the subject's eye, wherein the subject's eyefluoresces in response to light having a wavelength of about 600 nm toabout 900 nm. In some aspects, the light can have a wavelength of about400 nm to about 900 nm

A method described herein can further comprise administering to thesubject an effective amount of a fluorescent compound, wherein thedetecting or measuring occurs at least one day after the administrationof the fluorescent compound. In some aspects, the detecting or measuringcan occur at least one day after administering to the subject aneffective amount of a fluorescent compound. In some aspects, the methodsdescribed herein can further comprise the step of detecting or measuringFAF in the eye of the subject. In some aspects, the methods describedherein can further comprise the step of correlating an FAF pattern tothe presence, absence, or amount of immune cells in the subject's eye.In some aspects, the detecting or measuring can comprise performingcSLO, FAF, DNIRA, OCT, or OCTA, and correlating these modalities to thepresence, absence of amount of immune cells in the subject's eye. Insome aspects, the immune cells can be cells of the subject's innateimmune system and/or macrophage and/or microglial cells, dendriticcells, monocytes, mononuclear phagocytes, phagocytic immune cells.

DNIRA

In various aspects, system 100 can use optical imaging, using varioustechniques. For example, such techniques include, but are not limited tofundus photography, cSLO, FAF, angiography, OCT, OCTA, including threedimensional reconstructions of such. In various aspects, exposing an eyeto light comprises performing cSLO, FAF, DNIRA, angiography or OCT. Invarious aspects, the imaging is DNIRA. In various aspects, combinationsof any of the above techniques may be used.

The inventor previously demonstrated that following systemic delivery ofsodium iodate, patches of hypofluorescent FAF are not observed in vivoin areas of RPE loss (in the non-clinical setting, i.e., non-human eye)as would be predicted from clinical investigation (data not shown).However, by pre-labeling the RPE with a fluorescent dye, such as thenear infra-red (NIR) dye indocyanine green (ICG), a technique calledDelayed Near InfraRed Analysis (DNIRA), the RPE is made visible usingcSLO imaging. Once labeled, areas of RPE loss become apparent asquantifiable patches of hypofluorescence similar to those observedclinically. In various aspects, NaIO₃, FAF and DNIRA may be usedtogether to show, by way of example, the relationship between RPE loss,macrophages, macrophage polarization, and regulation of the M1 response.

For DNIRA, a compound suitable for fluorescence detection including anear-infrared (NIR) dye, such as, ICG when given at non-toxic doses, canlabel the RPE and therefore make it visible when viewed using the ICGexcitation/emission filters in the days or weeks thereafter.Importantly, this visualization in the days and weeks thereafter may bewithout re-administration of dye. Accordingly, in some aspects, acentral component of the DNIRA technique lies in its timing. This isdistinct from the present usage of ICG or other angiographic dyes thatare viewed immediately after injection, during the transit phase, or inthe immediate minutes to hours following injection, to determine theintra-vascular localization of dye and its immediate extravasation.

In some aspects, DNIRA can be used in a laboratory animal. In oneaspect, DNIRA may involve administration of a compound suitable forfluorescence detection, by way of non-limiting example, ICG (and,optionally, angiography) at about one or more days prior toadministration with a toxin or other agent that causes patchy geographicareas of RPE loss (e.g. NaIO₃) and optionally followed by, at about 1 ormore days (or about one week, or about one month, or about threemonths), an additional amount of NaIO₃ or another agent that causesexpansion of the areas of patchy RPE loss. For example, the otherchallenge that causes geographic atrophy expansion (e.g. as an initial,or second, or third, or fourth administration) may be a modulator ofcell survival, cell death, survival, autophagy, proliferation,regeneration, and the like.

In various aspects, the DNIRA technique can be used in a human subject.For example, DNIRA in a human subject may not comprise the use of atoxin. DNIRA in a human subject may comprise the evaluation of normal ordisease-associated changes in the eye, using a fluorescent dye, with theexcitation/emission filters in place but no angiography and following adelay of hours or days after dye administration.

Expansion of geographic atrophy is a U.S. Food and Drug Administration(FDA) acceptable primary outcome for clinical trial design. Accordingly,methods and systems as described herein can make possible observation ofgeographic atrophy, in particularly the expansion of geographic atrophy,in an animal model, thus permitting correlation between pre-clinicaldisease models and clinical trial design. The inability to clearlyidentify the geographic atrophy, or expansion of geographic atrophy, inan eye of an animal has precluded direct correlation betweenpre-clinical studies and clinical observation. Further, in some aspects,a system as described herein can allow for clinical evaluation of thesize and rates of expansion of geographic atrophy, including theexpansion of geographic atrophy, in a human subject.

In some aspects, the compound suitable for fluorescence detection can besuitable for imaging with various wavelengths of fluorescence. In someaspects, these wavelengths range from visible light to infrared, e.g.,390 nm to 1 mm, including, for example, blue light, white light, andnear-infrared. In some aspects, the dye is a near-infrared dye. In someaspects, the dye is ICG.

In some aspects, DNIRA can be performed (and/or delayed near infraredfluorescence (DNIRF) is observed) at about 6 hours, 12 hours, 24 hours,or about 2 days, or about 3 days, or about 4 days, or about 5 days, orabout 6 days, or about 7 days, or about 10 days, or about 14 days, orabout 21 day after the administration. In some aspects, the DNIRA can beperformed at least 1 day after the administration, or at least 2 days,or at least 3 days, or at least 4 days, or at least 5 days, or at least6 days, or at least 7 days, or at least 10 days, or at least 14 days, orat least 21 days after the administration. In some aspects, the DNIRAcan be performed at least about 2 hours, 4, hours, 8 hours, 24 hours, orat least about 7 days, or at least about 30 days, or at least 60 days,or at least 90 days after administering. In some aspects, the DNIRA maynot performed during the transit stage (e.g. the first passage of dye asit flows through the ocular blood vessels and into the ocular tissue) orminutes thereafter. In some aspects, angiographic imaging may not berequired, thus further distinguishing current dye-based imaging systemsfrom DNIRA.

In some aspects, the visualization can be effected using a cSLO. In someaspects, the visualization can be effected using white light andappropriate filters. In some aspects, the ICG excitation/emissionfilters can be 795 nm (excitation)/810 nm (emission) filters. In somecases, the visualization can be effected using a fundus camera or otherocular imaging device with appropriate spectra.

The RPE is a critical epithelial monolayer that serves a “nurse-cell”function for an eye's specialized photoreceptors, the rods and cones.Ocular diseases or disorders, such as, for example, AMD and RPD, are,without wishing to be bound by theory, causally linked in part toabnormalities of the RPE. DNIRA makes it possible to clearly identifythe RPE layer in vivo in an eye of an animal. Further, the leadingtechnique used to detect the RPE in the human eye, FAF, is ineffectiveor poorly effective in the rodent eye (by way of non-limiting example),possibly owing to a relative paucity of fluorophores such as lipofuscin.FAF imaging in the human eye is performed using the blue spectrum ofnon-coherent light in the presence of stimulation/emission filters, orcoherent blue light, and can identify areas of absent RPE (e.g.hypo-fluorescent signal) or abnormal RPE (e.g. hyper-fluorescentsignal). The inability to clearly identify the RPE in an eye of ananimal, in the absence of DNIRA, has precluded direct correlationbetween pre-clinical studies and clinical observation.

Accordingly, in various aspects, methods to make visible the RPE layer,such as, for example, DNIRA, in an eye of an animal for pre-clinicalinvestigation of ocular diseases or disorders are provided. Further, asdescribed herein, DNIRA, or variations thereof, can allow forvisualization of fluorescent immune cells in the eyes of an animal.Further, as described herein, DNIRA, or variations thereof, can allowfor visualization of fluorescent immune cells in the eyes of humansubject. In some aspects, the practicing of a method described hereinwith a human subject may not comprise toxin administration.

In some aspects, DNIRA can be used in the identification of an agentthat is effective for treating an ocular disease or disorder. In someaspects, DNIRA can be used as a method to evaluate a subject that has,or may have, an ocular disease or disorder (including, withoutlimitation AMD and RPD). In some aspects, DNIRA can be a surrogatebiomarker for diagnosis and/or prognosis and/or progression of an oculardisease or disorder (including, without limitation AMD and RPD). Forexample, DNIRA may be used to identify patterns, including lacy,reticular or leopard-like pattern of alternating hyper- andhypo-fluorescent DNIRA that is not seen in other imaging modalities,that are indicative of a ocular disease state (without limitation AMDand RPD). DNIRA may also be used to identify, and quantify, areas ofhyper- and hypo-fluorescent DNIRA.

In various aspects, DNIRA can be used to identify hypofluorescentfeatures of an eye. For instance, these areas appear black when imagedand therefore allow for easy quantitation (in contrast to ICG imaging,or in contrast to hyperfluorescent signal, which is grey-scale ratherthan black/white). Detection of hypofluorescent DNIRA, in some aspects,can be predictive of damaged or dead RPE. For example, hypofluorescentDNIRA may indicate one or more of an absence of RPE, abnormal/unhealthyRPE (which is unable to uptake ICG dye), RPE that does not lie incontact with Bruch's Membrane (and so are no longer in a position totake up ICG dye from the choroidal vasculature), and the presence oflipid that could be located either between the RPE and BM (thus blockingICG uptake), or could be internal to the RPE (thus blocking the RPEsignal).

In various aspects, DNIRA can be used to identify hyperfluorescentfeatures of an eye. For instance, these areas appear light when imagedand therefore allow for easy quantitation. Detection of hyperfluorescentDNIRA, in some aspects, is predictive of macrophages, including M1and/or M2 macrophages.

In various aspects, DNIRA can be used biomarker for diagnosis of anocular disease state (without limitation AMD and RPD) and promptsfurther evaluation and/or treatment with one of more agents, includingwithout limitation those described herein. In various aspects, DNIRA canbe used as a biomarker for prognosis of an ocular disease state (withoutlimitation AMD and RPD) and prompts further evaluation and/or treatmentwith one of more agents, including without limitation those describedherein. In various aspects, DNIRA can be used to improve identificationof suitable subjects for study recruitment and to evaluate progressionof disease. In various aspects, DNIRA can be used to monitor diseaseprogression.

In various aspects, DNIRA can be used to identify regions of diseasethat may be amendable to gene therapies, stem/progenitor or other celltherapies, or combined gene/cell therapies. By way of non-limitingexample, regions suitable for rescue by cell replacement or trophicfactor support, can be identified and targeted for therapy

In various aspects, DNIRA can be used as a companion diagnostic to anyof the agents described herein. In various aspects, DNIRA can be used toevaluate subject response to any of the agents described herein(including evaluating the effectiveness of any of the agents describedherein and/or the likelihood of response to any of the agents describedherein). In various aspects, the use of DNIRA can entail entranceinclusion or exclusion criteria, or endpoint analysis for clinical trialdesign.

In various embodiments, the present relates to methods for evaluating abinding eye disease using the systems and methods described herein. Forinstance, in various embodiments, there is provided a method forevaluating a subject's eye using DNIRA (e.g. using the systems andmethods described herein). In various embodiments, there is provided amethod for evaluating subject's eye by administering a fluorescentcompound, such as is indocyanine green (ICG), which is ingested by oneor more cells (including, without limitation, retinal pigment epithelial(RPE) cells and cells of immune system such as macrophages) and exposingthe eye to light having a wavelength of about 600 nm to about 900 nm,wherein the exposing occurs at least 24 hours (e.g. about 24 hours, 36hours, or about 2 days, or about 3 days, or about 4 days, or about 5days, or about 6 days, or about 7 days, or about 2 weeks, or about onemonth) after administering the fluorescent compound; and evaluating thefluorescent pattern in the eye for one or more features indicative of abinding eye disease or stage thereof. The features indicative of ablinding eye disease may be selected and/or identified based on analysisof a cohort of patients (e.g., comprising an experimental group havingthe blinding eye disease or stage thereof and a comparator group thatdoes not have the blinding eye disease). Features may be selected and/oridentified using pattern recognition and/or cluster analysis. Thepresence of the features in the subject's images may be determined usinga classifier algorithm, which may be trained using the images andanalysis from the cohort(s).

Such methodology allows for binning or classifying of subjectfluorescent pattern information in a manner that guides clinical trialdesign, e.g. directing enrollment or exclusion of patients intotherapeutic studies of a binding eye disease. For instance, suchmethodology, in various embodiments, allows for discriminating between asubject having a binding eye disease and a subject not having a bindingeye disease. Further, in various embodiments, such methodology allowsfor discrimination among or between subjects having a blinding eyedisease, e.g. for disease stage or progression, disease subtype, relatedmolecular genetics, pathobiology, drug response, etc. Suchdiscrimination and resultant direction of clinical trial decision-makingis described further elsewhere herein. Such discrimination and directionof clinical treatment are consistent with customized or personalizedclinical trial design or personalized medicine.

Further, in the presence of potential or various treatments, suchmethodology allows for binning or classifying of subject fluorescentpattern information in a manner that guides clinical decision-making,e.g. directing treatment of a blinding eye disease, if necessary. Forinstance, such methodology, in various embodiments, allows fordiscriminating between a subject having a blinding eye disease and asubject not having a blinding eye disease. Further, in variousembodiments, such methodology allows for discrimination among or betweensubjects having a blinding eye disease, e.g. for disease stage orprogression, disease subtype, related molecular genetics, pathobiology,drug response, etc. Such discrimination and resultant direction ofclinical decision-making is described further elsewhere herein. Suchdiscrimination and direction of clinical treatment are consistent withpersonalized medicine.

In various embodiments, the present methods allow for a comparisonbetween eyes in a test group and eyes in a control group (e.g. nothaving the blinding eye disease) to train an algorithm for classifying asubject's images for the presence of a binding eye disease or the statusor stage of the binding eye disease (e.g., as progressing or regressingin response to therapy (in cases where the method is being used formonitoring)). For instance, in various embodiments, the present methodsallow for the assembly of a database of various patterns or features ofeyes in a cohort, such a cohort comprising groups selected fromdiseased, non-diseased, early-stage disease, late-stage disease, diseasesubtypes, etc. Accordingly, in various embodiments, the images arecompared to allow for feature or pattern extraction, which can form thebasis of the classifier. In various embodiments, pattern recognitionusing a machine learning methodology is used to continually enhance thepower of the classifier as further images are analyzed, and which canprovide for continual improvement of the power to discriminateconditions for clinical decision-making (e.g. by providing objectivedecision support tools to assist medical professionals in diagnosis andprognosis of blinding eye conditions, and to assist researchers inclinical trial design.

In various embodiments, an image profile of the various eyes, whenanalyzed, allows for extraction of various features which areinformative about, for instance, the presence, absence, extent, subtype,etc. of a blinding eye disease. Image profile analysis and/or featureextraction can employ any suitable algorithm. Such an algorithm mayclassify a sample between blinding eye disease-afflicted andnon-afflicted groups. For example, samples may be classified on thebasis of imaging features as described herein, e.g. in subjects having ablinding eye disease, or suspected of having a blinding eye disease,versus a non-blinding eye disease population (e.g., a cohort from thegeneral population or a patient cohort with diseases other than ablinding eye disease) or versus other blinding or non-blinding eyedisease populations (e.g. a cohort with a particular blinding eyedisease compared against a distinct binding or non-blinding eyedisease). Various classification schemes are known for classifyingsamples between two or more groups, including Decision Trees, LogisticRegression, Principal Components Analysis, Naive Bayes model, SupportVector Machine model, and Nearest Neighbor model. Further, severaldifferent algorithms may be utilized for analysis of the eye imagingdata. The algorithms may include, for example, a machine learning singleclass classifier or an anomaly detector algorithm. As shown above,detection algorithms may be based on a support vector data description(SVDD). Additional algorithms may include a support vector machine(SVM), a relevance vector machine (RVM), a neural network, neuralanalysis, a large margin classifier, a kernel based classifier, aclassifier based on a probability density function (PDF) estimator, aclassifier based on a Parzen PDF estimator, a Bayesian classifier, aConstant False Alarm Rate (CFAR) detector, a fuzzy logic basedclassifier, and/or similar detection algorithms. In addition, thepredictions from multiple models or algorithms can be combined togenerate an overall prediction.

In various embodiments, the present methods of evaluation employ machinelearning and computational intelligence techniques, such as deep neuralnetworks, and combinations of supervised, semi-supervised andunsupervised learning techniques. Such machine learning andcomputational intelligence techniques provide, in various embodiments,image classification, image analysis, computer-aided diagnosis andprognosis, and/or pattern recognition information.

In various embodiments, the present methods of evaluation employ asupervised algorithm (by way of non-limiting example, linear region,random forest classification, decision tree learning, ensemble learning,bootstrap aggregating, and the like). In various embodiments, thepresent methods of evaluation employ a non-supervised algorithm (by wayof non-limiting example, clustering or association).

In various embodiments, the present systems and methods enable thepattern recognition, e.g. of various features of relevance to blindingeye diseases as described herein.

In various embodiments, discriminant analysis and/or classificationanalysis allows for the binning of patients so as to inform on an eyestatus (e.g. presence or absence of disease, severity of disease,identity of disease, and the like).

In various embodiments, the present systems and methods are useful inthe interpretation of various eye features as shown elsewhere herein.

In various embodiments, the present systems and methods are useful inthe detection of hypofluorescent patterns including presence, absence,or extent of, by way of non-limiting example, regions of profoundhypofluorescence (PHoF), intermediate hypofluorescence (IHoF), lowhypofluorescence (LHoF) (e.g. such levels of hypofluorescence may beassessed relative to image standards, e.g. a reduction of fluorescenceto about 90% or more of the hypofluorescence measured at the optic nervehead (ONH) (e.g. 60-90% of the ONH, 20-60% of the ONH), that may havesharp (readily delineated) borders or soft (blurry or fuzzy) borders,and complex fluorescent patterns.

In some embodiments, systems and methods described herein detect andidentify and classify hypofluorescent patterns and regions havingvarious hypofluorescent shapes. The detection of hypofluorescentpatterns may include, by way of example: detection and classificiationof a region of repeating hypofluroescent shapes into classes, which maybe pre-defined or generated automatically by way of a learningalgorithm. Such classes may correspond, for example, to various observedpatterns. Such patterns may includes, for example, in a “loose weave”pattern; a region of concentrated repeating hypofluorescent shapes, forexample, in a “tight weave” pattern; patterns of generally elliptical,oval or oblong shaped hypofluorescent regions, for example,hypofluroescent shapes resembling “fingerling potatoes”; patterns ofgenerally round or rosette shaped hypofluorescent regions, for example,hypofluorescent shapes resembling “leopard spots”; and regions ofintermediate hypofluroescence (IHoF) or low hypofluorescence (LHoF), forexample, resembling a “grey smudge”.

In various embodiments, the present systems and methods are useful inthe detection of hyperfluorescence patterns including presence, absence,or extent of, by way of non-limiting example, regions of profoundhyperfluorescence (PHrF), intermediate hyperfluorescence (IHrF), lowhyperfluorescence (LHrF), that may have sharp (readily delineated)borders or soft (blurry or fuzzy) borders or of “bright dots”, “brightspots”, “large bright spots” and various complex patterns including“haloes”.

In various embodiments, the present systems and methods are useful inthe detection of complex 2D patterns, by way of non-limiting example,labeled “tight weave”, “loose weave”, “grey smudge”, “oil stains”,“bullet holes”, etc.

In various embodiments, the present systems and methods allow for thedetection of the transition from the presence of drusen or pseudodrusenin the absence of geographic atrophy, macular atrophy (MA), or choroidalneovascularization, to the presence of drusen or pseudodrusen in thepresence of geographic atrophy, macular atrophy, or choroidalneovascularization. In various embodiments, the present systems andmethods allow for the detection of drusen regression (e.g. a reductionof drusen by about 25%, or about 50%, or about 75%, or about 100%).

In various embodiments, the present systems and methods allow for thedetection of ocular tissue loss (e.g. a reduction of ocular tissue lossby about 25%, or about 50%, or about 75%, or about 100%).

In various embodiments, the present systems and methods allow fordetection of presence, absence, or extent of drusen, pseudodrusen,picsiform lesions, inflammatory infiltrates, subretinal fluid, variablyshaped hyper- or hypo-pigmented regions (e.g. bull's eye lesion) in theabsence of geographic atrophy, macular atrophy, or choroidalneovascularization, to the presence of drusen, pseudodrusen, picsiformlesions, inflammatory infiltrates, subretinal fluid, variably shapedhyper- or hypo-pigmented regions (e.g. bull's eye lesion) in thepresence of any of any tissue loss, geographic atrophy, macular atrophy,or choroidal neovascularization.

In various embodiments, the present systems and methods allow fordetection of the total number of drusen, the size of drusen, thepresence of soft confluent drusen, hyperpigmentation, RPE detachments,hypopigmentation, hypo-reflective subretinal material, hyper-reflectivedots, subretinal basal laminar and/or basal linear deposits as depictedin Example 22 and FIG. 40.

In various embodiments, the present systems and methods allow fordetection of the extent (utilizing by way of non-limiting example, thearea, square root of the area, perimeter, diameter of best-fit circle)of regions of hypofluorescent DNIRA signal that coincide with high-riskfeatures such as the total number of drusen, the size of drusen, thearea of drusen, the volume of drusen, the border of drusen, the presenceof soft fuzzy drusen, the presence of soft confluent drusen,hyperpigmentation, RPE detachments, hypopigmentation, hypo-reflectivesubretinal material, hyper-reflective dots, subretinal basal laminar orbasal linear deposits, and the like. An illustration of this isexemplified in FIG. 40 and depicted in Example 22.

In some embodiments, the methods provide for pattern recognition in thefluorescence of the RPE and outer retinal layer in a subject, includingpatterns based on the loss of fluorescence of the RPE and outer retinallayer. RPE and outer retinal layer fluorescent patterns can be used fortraining classifier algorithm for evaluating blinding eye disease.

Accordingly, in various embodiments, the present provides ophthalmicimage-based biomarkers to allow for eye evaluation. In some embodiments,such ophthalmic image-based biomarkers are to be used alone or inconjunction with other ophthalmic image-based biomarkers (such as butnot limited to color, FAF, IR, NIR, OCT, OCT-A, angiography) ornon-image-based biomarkers (such as, but not limited to, genomics,tissue biochemistry, functional measures of disease, and other outcomesmeasures such as QALY (Quality Adjusted Life Years)) to enhance thediagnosis of disease, to identify previously known and unknown subtypesof disease, to identify patients likely to develop disease, to identifypatients likely to suffer disease progression, to identify patientslikely or not to respond to a particular intervention (e.g. therapy),and/or to identify patients whose future outcome or whose safety may becompromised by disease progression, disease intervention orenvironmental influence.

In various embodiments, the present provides methods for identifyingpatients with blinding eye disease earlier in a diseasecourse/progression. If detected early, earlier intervention may bepursued. Alternatively, the present provides methods for identifyingpatients with blinding eye disease later in a diseasecourse/progression. If detected later, less burdensome intervention maybe pursued.

In various embodiments, the present provides methods of detectingpatients more likely to progress from earlier disease to later disease(e.g. prognostic biomarker)

In various embodiments, the present provides method for measuring ormonitoring the rate of disease progression or the effect of anintervention over time (e.g. disease response) to one or moretherapeutic agents (e.g. a prognostic, predictive or monitoringbiomarker)

In various embodiments, the present provides a method of determiningpatient prognosis, e.g., to identify the likelihood of a clinical event,disease recurrence, or disease progression in patients who have blindingeye disease such as AMD, RPD, maculopathy, central serous retinopathy,uveitis, inherited retinal degeneration and the other disease mentionedelsewhere herein.

In various embodiments, the present provides a method for predicting theresponse of a patient or group of patients to an intervention (e.g. atherapy) to identify individuals who are more likely than similarindividuals without the biomarker to experience a favorable orunfavorable effect from exposure to a medical product or anenvironmental agent. For instance, such methods predict that a patientmay be a responder to a therapy. If so, in various embodiments, such apatient may be directed to receive treatment with the therapy.Alternatively, such methods predict that a patient may not respond to atherapy. If so, in various embodiments, such a patient may be directedto not receive treatment the therapy and, therefore, may be directed toalternative therapies. (E.g. a prognostic, predictive, monitoring orsafety biomarker).

In various embodiments, the present provides a method for demonstratingthat a biological response has occurred in an individual who has beenexposed to a medical product or an environmental agent. (e.g. apredictive or safety biomarker)

In various embodiments, the present provides a complimentary biomarkerto drive clinical trial design. In various embodiments, the presentprovides a companion biomarker to drive clinical trial design. Invarious embodiments, the present systems and methods allow for theselection of patients likely to respond to a clinical trial drug orother clinical trial intervention and therefore allow recruitment of asuitable population for a clinical trial.

In various embodiments, the present provides a method for personalizeddrug development or personalized medicine. For instance, the presentmethods allow, in various embodiments, interventions and/or productsbeing tailored to the individual patient based on their predictedresponse or risk of disease.

By way of illustration, in various embodiments, the present methodsprovide for diagnostic biomarkers or methods. For example, the presentmethods and systems identify complex 3D patterns of reducedhypofluorescent signal in patients with a family history of a blindingeye disease (e.g. AMD) but without a personal clinical diagnosis. Assuch, DNIRA may enable the early diagnosis of patients, or a subset ofpatients, likely to develop disease. By contrast, patients with no knownfamily or personal history do not have the same pattern ofhypofluorescence. Such information directs ready intervention. Invarious embodiments, the present diagnostic biomarkers or methods mayenable the early diagnosis of patients which have family or personalhistory of a blinding eye disease (e.g. AMD) but lack knowledge of such.Accordingly, such methods may direct early treatment in patients thatwould otherwise not be treated.

In various embodiments, the present methods inform treatment with one ormore therapeutic agents, such as, without limitation, an anti-VEGFagent, an ACE inhibitor, a PPAR-gamma agonist or partial agonist, arenin inhibitor, a steroid, and an agent that modulates autophagy, aswell as a semapimod, a MIF inhibitor, a CCR2 inhibitor, CKR-2B, a2-thioimidazole, CAS 445479-97-0, CCX140, clodronate, aclodonate-liposome preparation or gadolinium chloride.

Non-limiting examples of anti-VEGF agents useful in the present methodsinclude ranibizumab, bevacizumab, aflibercept, KH902 VEGF receptor-Fc,fusion protein, 2C3 antibody, ORA102, pegaptanib, bevasiranib,SIRNA-027, decursin, decursinol, picropodophyllin, guggulsterone,PLG101, eicosanoid LXA4, PTK787, pazopanib, axitinib, CDDO-Me, CDDO-Imm,shikonin, beta-, hydroxyisovalerylshikonin, ganglioside GM3, DC101antibody, Mab25 antibody, Mab73 antibody, 4A5 antibody, 4E10 antibody,5F12 antibody, VA01 antibody, BL2 antibody, VEGF-related protein,sFLT01, sFLT02, Peptide B3, TG100801, sorafenib, G6-31 antibody, afusion antibody and an antibody that binds to an epitope of VEGF.Additional non-limiting examples of anti-VEGF agents useful in thepresent methods include a substance that specifically binds to one ormore of a human vascular endothelial growth factor-A (VEGF-A), humanvascular endothelial growth factor-B (VEGF-B), human vascularendothelial growth factor-C (VEGF-C), human vascular endothelial growthfactor-D (VEGF-D) and human vascular endothelial growth, factor-E(VEGF-E), and an antibody that binds, to an epitope of VEGF.

In some embodiments, the anti-VEGF agent is the antibody ranibizumab ora pharmaceutically acceptable salt thereof. Ranibizumab is commerciallyavailable under the trademark LUCENTIS. In another embodiment, theanti-VEGF agent is the antibody bevacizumab or a pharmaceuticallyacceptable salt thereof. Bevacizumab is commercially available under thetrademark AVASTIN. In another embodiment, the anti-VEGF agent isaflibercept or a pharmaceutically acceptable salt thereof. Afliberceptis commercially available under the trademark EYLEA. In one embodiment,the anti-VEGF agent is pegaptanib or a pharmaceutically acceptable saltthereof. Pegaptinib is commercially available under the trademarkMACUGEN. In another embodiment, the anti-VEGF agent is an antibody or anantibody fragment that binds to an epitope of VEGF, such as an epitopeof VEGF-A, VEGF-B, VEGF-C, VEGF-D, or VEGF-E. In some embodiments, theVEGF antagonist binds to an epitope of VEGF such that binding of VEGFand VEGFR are inhibited. In one embodiment, the epitope encompasses acomponent of the three dimensional structure of VEGF that is displayed,such that the epitope is exposed on the surface of the folded VEGFmolecule. In one embodiment, the epitope is a linear amino acid sequencefrom VEGF.

In various embodiments, the present methods inform treatment with one ormore therapeutic agents, not limited to, a vitamin supplement, acomplement inhibitor or activator, a visual cycle modulator, an amyloidblocker or inhibitor, a neuroprotectant, an autophagy inhibitor, ananti-inflammatory, a modulator of macrophage behavior or activity, amodulator of CD36, a tyrosine kinase inhibitor, an RNA modulator, genetherapy or cell therapy. Examples of these include APL-2, Eculizumab,LFG316, Sirolimus, Fluocinolone acetonide (Illuvien), Fenretinide,Emixustat, Trimetazidine, Alprostadil, Moxaverine, Sildenafil, MC-1101(MacuClear), OT551, Tandospirone, GSK933766, Lampalizumab, ARC-1905,RNG6, Oracea, Brimonidine Tartrate, FMX-103, CLG-561, BIO-201, BIO-203,tesidolumab, unoprostone isopropyl, GS-030, ICR-14967, methotrexate,ONL-1204, RST-001, TT-231, KSI-401, OCU-100, RC-1 alpha, ACU-02,ACU-3223, alprostadil, AVT-2, HMR-59, INO-6001, INO-6002, MCT-355,NT-501, SIR-1046, SIR-1047, SIR-1076, zinc monocysteine, EBI-028, EG-30,RNA-144101.

In various embodiments, the present methods inform treatment with one ormore therapeutic agents, such as, a complement inhibitor or activator, avisual cycle modulator, an amyloid blocker or inhibitor, aneuroprotectant, an autophagy inhibitor, an anti-inflammatory, and amodulator of macrophage behavior or activity.

In various embodiments, the present methods provide for identificationof disease subtypes, e.g. amongst patients with a known diagnosis of ablinding eye disease (e.g. AMD).

Presently, the classification of AMD is limited to early (drusen), late(which can be wet (neovascular) or dry (with GA)). The co-existence ofvariably sized drusen (small, medium and large) and pigment, seen inclinical examination and color fundus photography, can further stratifyearly disease into subcategories of Low Risk and Intermediate Risk (orhigh-risk Early). However, given the multigenicity of AMD, with over 30genes and 200 SNPs, this represents significant simplification andprecludes the development of broad, targeted or subtype-specific, e.g.customized therapies (through over-inclusive clinical trial design) andultimately personalized therapies. Further, novel, rich image obtainedusing DNIRA, serving as image-based biomarkers with defined andill-defined features are particularly well-suited to the use ofcognitive computing for novel AI-derived pattern recognition. Coupledwith genetic, epigenetic and lifestyle data, DNIRA provides previouslyunknown correlations that point to efficient, customized clinical trialdesign and personalized therapeutics.

By way of example, subtypes in the context of wet AMD includesegregation by location of CNV (e.g. subfoveal, juxtafoveal,extrafoveal, and peripapillary) and/or traditional classifications (e.g.wholly classic, predominantly classic, minimally classic, and occult noclassic).

In various embodiments, the present methods find use in distinguishing(e.g. differentiating, discerning, perceiving) diseases of similarclinical appearance to assist with diagnostic accuracy and newdiagnostic classification. For instance, the present methods revealcomplex image-based patterns that are absent in patients with ocularconditions other than AMD, or frequently misdiagnosed as AMD.Reciprocally, the present methods can lead to correct diagnosis ofcertain diseases that otherwise would be diagnosed as AMD, e.g. but notlimited to, Stragardt disease, Adult Vitelliform disease, Bietti'scrystalline dystrophy, pattern dystrophy, and others. These otherconditions can be included in comparator cohorts to increase the powerof discrimination. Accordingly, in various embodiments, the presentmethods allow for improved diagnostic accuracy, enriched clinical trialenrollment, and directed, and a more appropriate, personalizedtherapeutic plan for an individual patient.

In various embodiments, the present methods find use in monitoringprogression of a blinding eye disease and/or the effect of a treatmentover time. Accordingly, in various embodiments, the present methods finduse as a monitoring biomarker. For example, by way of illustration, theability to measure dark (e.g. hypofluorescent) regions using the presentmethods makes it useful monitoring biomarker. Assessed serially overtime the present methods detect and measure the presence, status, orextent of a blinding eye disease such as AMD, or provide evidence of anintervention effect or exposure, including exposure to a medical productor an environmental agent.

In various embodiments, the present methods find use in quantifying orbetter qualitatively describing aspects of disease known to predictprogression (prognostic biomarker). In various embodiments, the presentmethods identify and quantify known high-risk features of disease suchas large, soft and confluent drusen. The AREDS 1 and 2 clinical studiesidentified risk factors that correlate with increased risk ofprogression from early to late AMD, and amongst these the most powerfulpredictor of progression is the presence of large, soft, confluentdrusen, with or without pigmentary change. Soft drusen can be identifiedduring clinical examination and are documented using color fundusphotography, but by nature has indistinct or “fuzzy” borders precludingtheir precise quantification even with alternative imaging methods suchas Optical Coherence Tomography (both cross-sectional and en face). Thepresent methods make detection and quantification of risk features suchas soft drusen possible and as such, can identify patients at risk,stratify risk of progression, and instruct which patients to enlist inclinical trials (particularly trials aiming to prevent late disease,either choroidal neovascularization, geographic atrophy, or macularatrophy).

In various embodiments, the present methods are used to quantify aspectsof disease currently hypothesized to predict progression (predictivebiomarker). In various embodiments, the present methods identify,quantify and follow over time the spatiotemporal changes in diseasefeatures predicted to correlate with disease but currently notquantifiable and hence not demonstrated. For example, in variousembodiments, the present methods can quantify total drusen load, changein the same over time, and the like.

Further, over time, For example, while large and medium drusen conferdisease risk and risk of progression, their dynamic changes aredifficult to observe. The present methods permit the establishment of aDynamic Drusen Index (DDI) that can calculate total drusen burden atsingle or multiple timepoints, or can calculate the change in particulardrusen subtypes. Relying on the current classification of drusen, theDDI is defined as: Total DDI (for all drusen types), Large ConfluentDynamic Drusen Index (LC-DDI), Large Dynamic Drusen Index (L-DDI),Medium Dynamic Drusen Index (M-DDI), and Small Dynamic Drusen Index(S-DDI). In various embodiments, DDI instructs which patients topreferentially enlist in particular clinical trials.

In various embodiments, the present methods identify presumptivemacrophages in the eyes of patients with diseases such as, withoutlimitation, the disease described herein (e.g. AMD, RPD, inflammatorydisease, inherited maculopathies, retinal degeneration, and oculartumors), thus identifying individuals who are more likely than similarindividuals without the biomarker to experience a favorable orunfavorable effect from exposure to a medical product such as macrophagemodulating therapy or environmental agent. For example, there arepresently no therapies available for the treatment of dry AMD despitethe current understanding that it is a disease of the innate immunesystem: no Phase III studies targeting the complement cascade have yetbeen successful. In various embodiments, the present methods allow forinvestigating the cellular arm of the innate immune system, e.g. themacrophage population (collectively all phagocytic immune cellsincluding but not limited to perivascular macrophages, microglia,resident parenchymal macrophages, dendritic cells and circulatingmonocytes) and/or the a quantification of a number of macrophages overtime, or with versus without treatment, etc.

In various embodiments, the present methods provide a method fordemonstrating that a biological response has occurred in an individualwho has been exposed to a medical product such as, but not limited to,bindarit, its derivatives and related compounds, macrophage modulatingagents, methotrexate, IkBa inhibitors and NFkB inhibitors, complementmodulators, other anti-inflammatory compounds or environmental agents,alone or in combination

In various embodiments, the present methods are used to bothqualitatively evaluate and quantify the pharmacodynamic response to suchtreatment, thereby being useful to establish proof-of-concept that adrug or other therapeutic produces the desired pharmacologic response inhumans thought to be related to clinical benefit, and to guidedose-response studies. In various embodiments, the present methods canfurther provide evidence of target engagement.

In various embodiments, the present methods are used to bothqualitatively evaluate and quantify the pharmacodynamic response to stemcell therapy—e.g. as a possible beneficial effect of macrophagemodulation in their survival and/or in use to restore damaged or missingRPE.

In various embodiments, illustrative blinding eye diseases relevant tothe present evaluation methods include, for example, dry AMD, wet AMD,reticular pseudodrusen (RPD), LORDs (late onset retinal degeneration),retinal degeneration associated with c1qTNF5 deficiency or itscorresponding gene mutation, or another maculopathy, including, but notlimited to, Stargart disease, pattern dystrophy, as well as retinitispigmentosa (RP) and related diseases. In one embodiment, the maculopathyis inherited. In other embodiments, the blinding eye disease relevant tothe present evaluation methods is an idiopathic disorder that may,without wishing to be bound by theory, be characterized by retinalinflammation, with or without accompanying macular degeneration,including, but not limited to, white-dot syndromes (e.g. serpiginouschorioretinopathy, serpiginous retinopathy, acute posterior multifocalplacoid pigment epitheliopathy (APMPPE), multiple evanescent white dotsyndrome (MEWDS), acute zonal occult outer retinopathy (AZOOR), punctateinner choroidopathy (PIC), and diffuse subretinal fibrosis (DSF)). Inother embodiments, the blinding eye disease relevant to the presentevaluation methods is central serous retinopathy (CSR). In otherembodiments, the blinding eye disease relevant to the present evaluationmethods is a retinopathy, including diabetic retinopathy.

In various embodiments, the present evaluation methods find use inevaluation of a tumor of the eye. For instance, in some embodiments, thepresent methods allow for the detection of macrophages, e.g.tumor-associated macrophages. For instance, the present methods allowthe detection of hyperfluorescent bright dots and/or a quantification ofhyperfluorescent bright dot density. Such dot density correlates withclinical classification (e.g. as determined by the observation/gradingof a clinician) and always for standardization of diagnosis and a lesssubjective classification scheme. In various embodiments, the presentmethods allow for detection or prediction of ocular tumor-associatedfluid, height of ocular tumor, and the like.

In various embodiments, the present evaluation methods find use inevaluation of a tumor of the eye, such as tumor being one or more of(e.g. a metastasis to the eye of) a basal cell carcinoma, biliary tractcancer; bladder cancer; bone cancer; brain and central nervous systemcancer; breast cancer; cancer of the peritoneum; cervical cancer;choriocarcinoma; colon and rectum cancer; connective tissue cancer;cancer of the digestive system; endometrial cancer; esophageal cancer;eye cancer; cancer of the head and neck; gastric cancer (includinggastrointestinal cancer); glioblastoma; hepatic carcinoma; hepatoma;intra-epithelial neoplasm; kidney or renal cancer; larynx cancer;leukemia; liver cancer; lung cancer (e.g., small-cell lung cancer,non-small cell lung cancer, adenocarcinoma of the lung, and squamouscarcinoma of the lung); melanoma; myeloma; neuroblastoma; oral cavitycancer (lip, tongue, mouth, and pharynx); ovarian cancer; pancreaticcancer; prostate cancer; retinoblastoma; rhabdomyosarcoma; rectalcancer; cancer of the respiratory system; salivary gland carcinoma;sarcoma; skin cancer; squamous cell cancer; stomach cancer; testicularcancer; thyroid cancer; uterine or endometrial cancer; cancer of theurinary system; vulval cancer; lymphoma including Hodgkin's andnon-Hodgkin's lymphoma, as well as B-cell lymphoma (including lowgrade/follicular non-Hodgkin's lymphoma (NHL); small lymphocytic (SL)NHL; intermediate grade/follicular NHL; intermediate grade diffuse NHL;high grade immunoblastic NHL; high grade lymphoblastic NHL; high gradesmall non-cleaved cell NHL; bulky disease NHL; mantle cell lymphoma;AIDS-related lymphoma; and Waldenstrom's Macroglobulinemia; chroniclymphocytic leukemia (CLL); acute lymphoblastic leukemia (ALL); Hairycell leukemia; chronic myeloblastic leukemia; as well as othercarcinomas and sarcomas; and post-transplant lymphoprol iterativedisorder (PTLD), as well as abnormal vascular proliferation associatedwith phakomatoses, edema (e.g. that associated with brain tumors), andMeigs' syndrome.

In various embodiments, the present evaluation methods find use inevaluation of a tumor of the eye, such as a primary intraocular cancer(e.g., without limitation, intraocular melanoma or primary intraocularlymphoma). In various embodiments, the present evaluation methods finduse in evaluation of retinoblastoma or medulloepithelioma.

Image Analysis Platform and Output

Systems described herein can provide output in the form of either rawimages for user analysis, or processed images for end-user (consumer)utilization. The output may be represented as visual elements on a GUIdisplay. In one aspect, the output can be comprised of raw imagesderived from the application of DNIRA to subjects with AMD and/or otherblinding diseases prior to their registration, alignment, enhancement,adjustment, segmentation, or any other relevant image processing,provided as single or composite images, randomly or serially-arranged.

In some aspects, the output can be comprised of processed imaged derivedfrom the application of DNIRA to subjects with AMD and/or other blindingdiseases after initial processing, that could include any or all ofregistration, alignment, enhancement, adjustment, segmentation, or anyother relevant image processing, provided as single or composite images,randomly or serially-arranged.

Systems and devices described herein can provide for image utilizationby independent users and/or for users through a centralized imageanalysis center or platform. Independent users include, but are notlimited to, physicians, vision care specialists, ophthalmic technicalstaff, vision scientists, contract research organizations, the drugdevelopment sector and those involved in clinical trial design andexecution. Centralized image platforms could include, by way ofnon-limiting example, cloud-based systems, web-based systems,network-based systems, and locally active network (LAN) based systems.

Systems and devices described herein can provide independent users,and/or centralized users with a graphical user interface (GUI)compatible with multiple systems for desk-top, lap-top, or hand-helddevices. Systems and devices described herein can provide centralizedusers with DNIRA image software for further analysis. DNIRA imagesoftware can be implemented for such analysis. A DNIRA image softwarecan capable of, but not limited to processes such as registration,alignment, contrast adjustment, sharpening, segmentation for structuredanalysis of pre-defined features, such as areas of hypofluorescentsignal, hyperfluorescent signal, hyperfluorescent dots, complex2-dimensional patterns.

Systems and devices described herein can provide provides centralizedusers with DNIRA image software capable of, but not limited to processessuch as registration, alignment, contrast adjustment, sharpening,segmentation for unstructured analysis of undefined or complex featuresother than those addressed by structured analysis.

Graphical User Interface

Systems described herein can provide an iteratively-improved graphicuser interface (GUI), intended to optimize day-to-day and specializeduser-guided image manipulation and analysis. Such GUIs could be madeavailable on hand-held devices, iPads, laptops, and other personalizedcomputing systems. Such GUIs could be embedded in commercially availableimaging equipment or purpose-specific devices optimized for DNIRA.

In an aspect, the GUI can comprise a series of visual elements thatdynamically update based on control commands and image output data asdescribed herein. The GUI may include for example, an image carousel,tiled array with enlarged images, grid or tiled array withoutenlargements.

Software-Embedded Chips and/or Software Programs for Dissemination

In some aspects, device-independent software programs can be developedthat can process DNIRA images acquired from any suitable imaging system(e.g. cSLO or non-cSLO based). In some aspects, the software program canbe made available for incorporation into current or future devices; inanother, the software program is made available to current device andsoftware manufacturers for post-acquisition image analysis.

Centralized Software System for Data Analysis and Iterative ProgramDevelopment

In some aspects, a system described herein can comprise centralizedimage processing, that could be for example, without limitation, becloud-based, internet-based, locally accessible network (LAN)-based, ora dedicated reading center using pre-existent or new platforms.

In some aspects, the software would rely on structured computation, forexample providing registration, segmentation and other functions, withthe centrally-processed output made ready for downstream analysis. Inthis case, output could be a single or series of adjusted images (e.g.registration, contrast, sharpness, etc. . . . ), provided to the user inrandom or non-random order, for subsequent use.

In a variation of this aspect, the software would rely on structuredcomputation, with the output pre-analyzed for the user and comparedagainst relevant modifiable or non-modifiable risk factors for disease.In this case, output could be a single or series of adjusted images(e.g. registration, contrast, sharpness, etc.), provided to the userwith a qualitative or quantitative description of the image content(s).

In some aspects, the software would rely on unstructured computation,artificial intelligence or deep learning, such that in addition toreadily-defined features (such as profoundly hypofluorescent/brightareas or hyperfluorescent dots), regions of variable grey-scale would beiteratively analyzed using layered or multi-layered processing. Asunstructured computation benefits from large and increasing numbers ofimages, unstructured DNIRA analysis would be particularly well-suited toa cloud-based, networked or reading center-based platform.

In a variation of this aspect, the software would rely on unstructuredcomputation, such that regions of high complexity could be iterativelyanalyzed using layered or multi-layered processing. By way ofnon-limiting example, these can include complex 2D patterns of varyinggrey-scale density. Output could be a single or series of adjustedimages (e.g. registration, contrast, sharpness, etc. . . . ), providedto the user with a qualitative or quantitative description of the imagecontent(s), potentially over time, and, for example, with and withouttreatment.

In a further variation of this aspect, the software would rely onunstructured computation, so-called “artificial intelligence” or “deeplearning”, such that in addition to readily-defined features (such asprofoundly hypofluorescent/bright areas or hyperfluorescent dots),regions of variable grey-scale would be iteratively analyzed usinglayered or multi-layered processing, however, in this case, output couldbe provided to the user with a qualitative or quantitative descriptionof the image content(s), potentially over time, and, for example, withand without treatment.

AMDI Phenotyping and Genotyping

Present systems for categorizing or describing subjects with AMD arelimited. For example, in a commonly-used system based on clinicalexamination and/or color fundus photographs, subjects are classified aseither early or late. All subjects with early disease are considered“dry”, and are commonly diagnosed by the presence of 5 or more drusenlarger than “droplets” (<10 μm). Late AMD can be exudative (neovascular,or “wet”), or non-exudative (atrophic, or “dry”), and both areassociated with vision loss; both forms of late disease can co-exist.Late “wet” AMD can co-exist with early dry or late dry.

A commonly used system uses a so-called “simple score” to identifysubjects at high-risk of progressing from early to late disease (eitherexudative or non-exudative). Accordingly, the presence of 2 or morelarge drusen (>125 μm) in one eye, or pigment clumps in one eye yields ascore of 1 each; the presence of both in both eyes therefore yields ascore of 4. Subjects with scores of 3 or more are more likely toprogress from early to late AMD. In some systems these subjects arecategorized as “high-risk early” or in some systems, as “intermediate”AMD.

In contrast to these simple classifications, AMD is known to be acomplex multigenic disorder with over 20 associated mutations/SNPs thatcan be inherited alone or in combination. Disease expression is furthermodified by epigenetic risk factors such as smoking, diet and nutrientsupplementation.

FAF can be a standard method for analysis of dry AMD, and regions ofprofoundly hypofluorescent/dark signal indicate the presence of GA andtherefore late disease. Complex 2D patterns of hyperfluorescent FAF canalso be observed and some, when found in conjunction with patches of GA,are associated with increased rates of patch expansion. These include,for example, the banded and diffuse (speckled) patterns. To date, therehas been no definitive correlation between the genetics (genotype) ofAMD and the clinical patterns (phenotype) determined by clinicalexamination or specialized imaging such as FAF or OCT.

AMDI can identify previously unknown image-based patterns of disease,e.g., provides novel phenotypic description of individual subjects.Accordingly, there is a need for image-based systems that can stratifysubjects into different subtypes of disease that may differ according torates of progression, susceptibility to epigenetic risk, response todifferent therapies and ultimately to rates of vision loss. AMDI candramatically increase an ability to define these groups. As AMDIprovides unprecedented description of different patterns or subtypes ofdisease, the results of a centralized imaging platform, iterativelyimproved used structured and unstructured learning, can then be comparedagainst GWAS analysis, epigenetic status, and other medical ordemographic information.

Biology of DNIRA

FIGS. 22 and 23 are graphical illustrations of biology of DNIRA signalin RPE and macrophages respectively. FIG. 22 depicts structural andfunctional aspects of DNIRA signal in RPE, where profoundlyhypofluorescent/black signal occurs in associations with severalpathological changes. Green arrows indicate movement/uptake of ICG dyefrom choroidal circulation to RPE monolayer. “Normal” levels of DNIRAsignal are illustrated as light green cells capable of internalizingdye. In the lower panel the normal RPE, Bruch's membrane (BrM) andchoroidal vasculature is illustrated, along with commonly observedchanges that occur in dry AMD. Hypofluorescence is observed inassociation with GA, RPE detachments, and drusen or other sub-RPEdeposits (basal linear, basal laminar deposits). A mid-grey signal isillustrated in regions where RPE are in their correct anatomicalposition relative to the choroidal blood vessels but where cellmetabolism is surmised to be abnormal. FIG. 23 depicts structural andfunctional aspects of DNIRA signal in macrophages. Hyperfluorescent dots(of a particular size, shape and motility) can appear in conjunctionwith the signals obtained in FIG. 22. Green arrows indicatemovement/uptake of dye from choroidal circulation to retinal/RPE layerwhere tissue macrophages are capable of internalizing it. In addition tothe fact that macrophages may internalize the dye directly in thecirculation before the cells themselves influx into the retinal/RPEtissue. During inflammation, increased phagocytic activity may correlatewith increased dye uptake and the generation of brightly fluorescentdots. Normally situated deep in the choroid (macrophages) or in theinner retina (microglia), their recruitment to the sites of RPE andphotoreceptor damage would also suggest the presence of bright DNIRAdots.

DNIRA can identify previously unknown image-based patterns of disease,e.g., provides novel phenotypic description of individual subjects.Accordingly, there is a need for image-based systems that can stratifysubjects into different subtypes of disease that may differ according torates of progression, susceptibility to epigenetic risk, response todifferent therapies and ultimately to rates of vision loss. DNIRA candramatically increase an ability to define these groups. As DNIRAprovides unprecedented description of different patterns or subtypes ofdisease, the results of a centralized imaging platform, iterativelyimproved used structured and unstructured learning, can then be comparedagainst GWAS analysis, epigenetic status, and other medical ordemographic information.

DNIRA as Functional Imaging

Without wishing to be limited by theory, it is understood that thepathogenesis of AMD derives in part from the inability to adequatelytransport oxygen and nutrients from the underlying choroidal bloodvessels to the RPE/photoreceptor (RPE/PhR) complex, which may beexacerbated as drusen or other deposits (e.g., lipids, basal laminardeposits, etc. . . . ) accumulate or as Bruch's membrane (thespecialized basement membrane of both the choroidal endothelium and RPEmonolayer) thickens. Likewise, the removal of metabolic waste from theRPE/PhR complex may also be impeded. A disturbance of this normalphysiological exchange is believed to underlie some aspects of disease.DNIRA represents the first measure of this transport system, and sorepresents the first functional imaging method. Other imaging methods todate are structural and do not rely on such dye uptake.

DNIRA relies on the circulation of systemically delivered dye throughthe choroidal vasculature, its passage through Bruch's membrane and itsactive uptake into the retinal pigment epithelium (RPE) layer. Areduction in dye uptake can therefore be caused in several ways: throughinterposition of material between the choroid and the RPE (such asdrusen or basal linear or basal laminar deposits, or deposits ofunspecific composition), the separation of the RPE from its basementmembrane (such as an RPE detachment, which may be serous, drusenoid,haemorrhagic or other), the localized loss of the choroidal bloodvessels (as occurs when tumours infiltrate the choroid) or, by a generaldysfunction of the RPE/Bruch's membrane complex that is unable, or oflessened ability, to transport the dye from the choroidal circulation tothe RPE/photoreceptor complex. If this ability is completely lost,regions of DNIRA appear black. If this ability is partially lost, areasof DNIRA may appear grey. Such grey regions may be a sensitive indicatorof decreased RPE health.

In the following two examples, the DNIRA signal is reduced when a smallgroup of choroidal tumour cells preclude blood flow in a small region.In the second example, regions of black DNIRA recover as an initiallylarge RPED resolves.

In one example of DNIRA as a functional imaging method, FIGS. 41-44 showa sequence of DNIRA images and OCT images of a patient's eyes obtainedover four sessions. Recovery of functionality can be observed as thepatient recovered from large RPE detachment and soft drusen diminished,with recovery in functionality corresponding to signal recovery(decrease in black pitch) in the DNIRA images over time.

A striking feature seen in FIGS. 41-44 is the reduction in 2D area ofthe regions of black, hypofluorescent DNIRA. This is not observed usingFAF, the current gold standard, wherein regions of black,hypofluorescence correspond with region of atrophy, and so, in theabsence of any available interventions, can only enlarge over time. Inthis instance, concurrently acquired and registered OCT images show thatthe return to normal levels of DNIRA fluorescence corresponds to areduction, or flattening, of the retinal pigment epithelium detachment(RPED) and concurrent resolution of drusen.

Significantly, DNIRA images can reveal aspects of dynamic functionalbehavior of retinal tissue and thus be used as a form of functionalimaging. In contrast, conventional imaging modalities such as FAF do notprovide dynamic functional behavior of retinal tissue as signal loss isobserved when tissue is dead and no recovery is available.

As a form of functional imaging, DNIRA can be used to measure treatmentresponse in clinical trials, such as trials of pharmacotherapy,nutraceutical, thermal therapy, light-based therapy, stem cell therapy,gene therapy, or the like.

As a form of functional imaging, DNIRA can be used to follow the courseof disease in a clinical (medical) setting

In another example of DNIRA as a functional imaging method, FIG. 45illustrates a DNIRA signal in black (as indicated by the arrow) wherethe tumour cells are blocking the flow of dye-labeled blood in thechoroid, so precluding its uptake into the overlying RPE andphotoreceptor tissue.

Taken together, the data in FIGS. 41-45 may suggest that DNIRA, thefeatures of DNIRA, the constellation of DNIRA features that make up aphenotype, the DNIRA classifier and DNIRA predictor can serve asbiomarkers to drive precision drug development and personalizedmedicine. This is of particular importance in jurisdictions wherefunctional endpoints are required for late phase clinical trial.

Dyes

While reference has been made to ICG dye, in some aspects, otherfluorescent compounds are substituted for ICG and are included in any ofthe aspects described above. Such fluorescent compounds can be suitablefor imaging with various wavelengths of fluorescence. In some aspects,these wavelengths range from visible light to infrared, e.g., 390 nm to1 mm, including, for example, blue light, white light, andnear-infrared. In some aspects, the dye can be a near-infrared dye.Further, while specialists in eye disease, particularly retinal diseasehave used ICG, and other dyes such as fluorescein, for the purposes ofangiography, angiography is not required for DNIRA.

Angiography, by definition “the study of blood vessels”, requires theintra-vascular injection of dye and evaluation of the changes that occurin the dye distribution thereafter, starting within seconds andextending to the minutes and hours thereafter, described by those versedin the art as leakage or staining, within defined time frames such ascirculation phase, transit phase, and re-circulation or late phase. AsDNIRA does not evaluate blood vessels or their integrity, dyes can beprovided by other routes of administration, such as oral and imagingperformed thereafter in hours or days or weeks.

In some aspects, the fluorescent compound can absorb light at awavelength of about 600 nm to about 900 nm and/or emits light at awavelength of about 750 nm to about 950 nm. In some aspects, fluorescentcompound can have the following features: about 795 nm (excitation) andabout 810 nm (emission).

Grey-Scale Feature Extraction

In some embodiments, processing of DNIRA images may including extractingfeatures using a Gray Level Co-occurrence Matrix (GLCM) analysis. Inaccordance with GLCM, a matrix is constructed by counting the number ofco-occurrences for a pixel value at different distances. This matrix isthen be used to extract statistical coefficients which are generallyreferred to as GLCM features, which include energy, entropy, contrast,homogeneity, correlation, shade, and prominence.

Classifier

As noted above, the computer systems described herein may be adapted toprovide a computer system that processes ophthalmic image data, such asocular images, to automatically classify data samples into one or moregroups. Each of such special purpose computer systems may be referred toas a “classifier” herein.

In some embodiments, a classifier may be configured to distinguish,stage, or grade variants of binding eye disease such as, but not limitedto, as Age Related Macular Degeneration (AMD), monogenic eye disease,inherited eye disease, inflammatory eye disease, CSR, serpiginous,ocular tumours or the like.

In some embodiments, a classifier may be configured to distinguish,stage, or grade variants of central nervous system (brain) disease, suchas dementia, Alzheimer's disease, or the like, wherein aspects orcomponents of the disease concurrently exist in eye and brain, such asthe deposition of amyloid.

In some embodiments, a classifier may be configured to classifyparticular regions of eye by tissue state such as, but not limited to,normal tissue, drusen, pseudodrusen, or GA. In some embodiments, a mapof tissue state across the eye is generated by sampling eye image dataat a plurality of regions and classifying the tissue state of eachregion. Such a map may be used for visualizing disease state, e.g., uponpresentation to a graphic user interface. Maps generated at differenttime periods for the same eye may be used to visualize diseaseprogression.

The maps may also be used as input data to other classifiers orconfigured to classify a patient, subject, or eye by, for example,extent of tissue loss, disease stage or grade. The input data may beused for such classifiers at a training phase or a classification phase.The maps may also be used as input data to a predictor, as detailedbelow, for predicting an area of disease progression (e.g., tissue lossor neovascularization) or a rate of disease progression or diseaseexpansion.

FIG. 46A depicts a classifier 1000 exemplary of an embodiment. Asdepicted, classifier 1000 may receive image data and non-image data asinputs and processes these data to produce a category indicator, namely,a digital indicator of the computed category, as an output. In oneexample, the category indicator may, for example, be indicative of apre-defined category. The classification indicator may, by way ofnon-limiting example, categorical, continuous, binary, or descriptive.

Image Data

The image data input to classifier 1000 may include multi-modal imagedata as described herein. In one embodiment, the image data includesdata produced at image acquisition 302 (FIG. 4). In another embodiment,the image data includes data produced at preprocessing 304 (FIG. 4). Inanother embodiment, the image data includes data produced atpostprocessing 306 (FIG. 4). Classifier 1000 may implement some or allof the preprocessing and postprocessing functions such as registration,segmentation, feature extraction, etc.

Imaging modalities of image data that is input to classifier 1000 may beat least one of delayed near-infrared analysis (DNIRA), infra-redreflectance (IR), confocal scanning laser ophthalmoscopy (cSLO), fundusautofluorescence (FAF), color fundus photography (CFP), opticalcoherence tomography (OCT), OCT-angiography, fluorescence lifetimeimaging (FLI), multispectral detection, and polarized fluorescenceimaging.

DNIRA image generation may be performed as described herein. Briefly,images are acquired using a commercially available confocal scanningophthalmoscope (e.g., Spectralis, Heidelberg Engineering, Germany) orother capable imaging device, before and at a suitable time after thesystemic administration of ICG dye. Additional images are also obtainedin the red-free, FAF (488/500 nm excitation/emission), IR reflectancechannel (830 nm) and ICG fluorescence channels (795/810 nmexcitation/emission), along with optical coherence tomography (OCT),optical coherence tomography angiography (OCT-A), fluorescence lifetimeimaging (FLI), color fundus photography, multispectral detection, andpolarized fluorescence imaging. Fluorescein or ICG angiography may ormay not be performed.

Non-Image Data

The non-image data input to classifier 1000 may include other datareflective of the biology or health of the patient including, forexample, age, height, weight, body mass index, blood pressure, measuresof bodily tissue or fluids, or to measures of their visual function,evaluated using methods such as visual acuity (VA), low luminance visualacuity (LLVA), delta LLVA, microperimetry, dark adaptometry, readingspeed, preferential looking tests, electroretinography, patientdemographic data, concurrent illness, medications, allergies, smokingstatus, or the like. The non-image data may be particular to a specificpatient or may be data reflective of a particular group of patients. Thenon-image data may include record data automatically retrieved from adatabase storing patient health records.

The non-image data may be particular to a specific patient or may bedata reflective of a particular group of patients. The non-image datamay include record data automatically retrieved from a database storingpatient health records.

Non-image data input to classifier 1000 may include data related tosystemic biomarkers. Such biomarker samples may include blood or cellsamples (e.g., buccal) for genetic testing, genomic testing, or mayinclude blood sampling for, by way of non-limiting example, cellularanalysis, protein analysis, proteomics, metabolomics, transcriptomicsanalysis, or may include samples of other bodily tissues such as gum andperiodontal sampling or urine or stool sampling, for by way ofnon-limiting example, evaluation of oral or gut microbiome

The non-image data input to classifier 1000 may also include parametersof a pattern recognition model. For example, when classifier 1000 isimplemented as a CNN, such parameters may include the CNN's kernels,template, window size, or the like.

Data Transfer

During data transfer, for example, during preprocessing and prior toinput to classifier 1000, data may be handled in compliance with privacyrights regulations, such as the Personal Health Information ProtectionAct (PHIPA) in Ontario, or other regions globally. Input data, includingboth image and non-image data, may be de-indentified and anonymizedmultimodal images with or without metadata and exported usingproprietary (e.g., Heyex, Heidelberg Engineering, Heidelberg, Germany)or non-proprietary software to local drives, network drives, distributednetworks, or a cloud computing service.

Data Storage

In some embodiments, storage of data that may be used as input forclassifier 1000 follows a hierarchical tree representation. Each treebranch contains information about session number, imaging modality,wavelength/dye used for that modality and finally the filename. The filename follows a specific pattern which contain the patient number sessionnumber and whether the image is for the left eye ‘OS’ (short for oculussinster) or right eye ‘OD’ (oculus dextrus, right eye).

In one specific embodiment, the image data may be input to classifier1000 in the form of a data structure as depicted in FIG. 47, for storingmultimodal retinal images.

A tree representation of data storage is illustrated in FIG. 47. Eachpatient is supplied with an identification (ID) number which formsparent branch in the data storage path. Within each patient branch thedata can be stored in the RAW, registered or further analysis datafolder. Within each of these folders the images are stored into a newbranch indicating their visit session number denoted by “S”. For eachsession, multiple images are acquired. The images are further dividedinto “Angiogram”, “Fundus”, “HRA”, and “OCT”; based on the modality ofacquisition.

The modality tree can branch further, for instance an angiogram branchcontains images related to angiography which are used to investigate theflow of blood in the back of the eye. Angiography can be performed usingdifferent agents such as ICG (ICGA) or Fluorescein (FA). The angiogrambranch is then divided to two new branches indicating the type of dyeused.

At the end of each branch, images are labelled using a precise patternwhich contains information about the ID, visit number, imagingwavelength and eye id. Both the hierarchical tree and the file name maybe used to easily identify and retrieve the needed images from a dataserver. For example, to retrieve the central fundus autofluorescence(FAF) image for patient 1 for the left eye, from their first visit, thefollowing path may be generated:001>RAW>S1>Fundus>HRA>Central>P008_S1_FAF_OD.tif

In some embodiments, for single or multi-site studies, data may beconfigured using a forest.

Machine Learning

In some embodiments, classifier 1000 may be configured to automaticallyidentify important features or phenotypes, or combinations of featuresor phenotypes during a training stage, as detailed below.

In some embodiments, classifier 1000 can train a pattern recognitionmodel 1002 that is one or more of a convolutional neural network (CNN),decision trees, logistic regression, principal components analysis,naive Bayes model, support vector machine model, and nearest neighbormodel.

A neural network may include, for example, three layers of neurons: aninput layer, a hidden layer and an output layer. The hidden layer is alinear combination of the input x and a bias (e.g., z=Wx+b). Neurons ateach layer may be activated via some nonlinearity (e.g., a=tanh (z)).The output layer is configured to generate an assessment of input layer(e.g., normal/GA), and errors are corrected via back propagation.

Convolutional neural networks may be used for image classification, andscan across an image with filters that pick out different patterns. Eachsuccessive layer picks out more complex patterns.

A convolutional neural network may be composed of two parts, featureextraction part and class identifier. The convolutional layers are knownas feature extractors. The first layers extract simple features such aslines and edges while the following layers create combinations of thesefeatures. In the final layers of a feature extraction part, a moreabstract features representation is created by a chain of combination ofcombinations of features. The second part of the convolutional neuralnetwork, the class identifier, identifies to which class these abstractrepresentations belong.

An example convolutional neural network architecture is shown in FIG.48. An input layer receives input data. A convolutional layer applies aconvolution operation to the input (emulating the response of anindividual neuron to visual stimuli), passing the result to the nextlayer. A pooling layer reduces the dimensions of the data by combiningthe outputs of neuron clusters at one layer into a single neuron in thenext layer. A fully connected layer connects every neuron in one layerto every neuron in another layer. The flattened matrix goes through afully connected layer to classify the images. An output layer outputs,for example, a classification.

A trained classifier identifies important features or combination offeatures (phenotypes) that help it learn a particular class. During atraining phase, classifier 1000 attempts to use many differentiterations of features combination until it identifies important sets offeatures, that may be a single feature or a plurality of features, thatit uses to determine the class.

In some embodiments, features may be clinically identified, such asgenerally elliptical shaped hypofluorescent regions resembling“fingerling potatoes”. For example, the classifier may process inputimage data corresponding to a greyscale region and, upon performingclassification, generate a classifier indicator in form of a categorylabel such as “fingerling potato”, “loose weave”, “tight weave”,“leopard spots”, and the like, as described herein.

Furthermore, examining the different layers of the CNN, may allow foridentification a new set of features or phenotypes, and their importancemay be examined in identifying the classes.

In some embodiments, a pattern recognition model may be implemented as aU-Net model built and trained, by way of non-limiting example, usingKeras with TensorFlow backend.

In Use

The operation of classifier 1000 is described with reference to theflowchart of FIG. 49.

At a training phase 1010, at block 1012 classifier 1000 may receivetraining data corresponding to image data and non-image data, asdescribed herein. In an example, training data may be image datacorresponding to a plurality of ocular images.

In some embodiments, the plurality of ocular images includes en faceimages and cross-section images.

In some embodiments, the plurality of ocular images of the training datacorrespond to a plurality of imaging modalities.

In some embodiments, the plurality of imaging modalities may be at leastone of delayed near-infrared analysis (DNIRA), infra-red reflectance(IR), confocal scanning laser ophthalmoscopy (cSLO), fundusautofluorescence (FAF), color fundus photography (CFP), opticalcoherence tomography (OCT), OCT-angiography, fluorescence lifetimeimaging (FLI), multispectral detection, and polarized fluorescenceimaging.

In some embodiments, input to classifier 1000 may be en face images fromthe central macular region, or multiple en face images from adjacent orextra-macular areas, which may or may not be acquired, assembled orstitched into a composite image.

At block 1120, classifier 1000 may perform feature extraction andfeature selection to generate features based on the training data tobuild a pattern recognition model 1002.

In some embodiments, a pattern recognition model may be built for eachof the plurality of imaging modalities.

At a classification phase 1020, at block 1022 classifier 1000 mayreceive image data, for example, a plurality of ocular imagescorresponding to a plurality of imaging modalities.

At block 1024, classifier 1000 may classify features of the image datausing pattern recognition model 1002. Classifier 1000 may then output,for example, a segmented image such as a masked image illustrating theidentified features, and a category indicator for one or more of theclassified features.

In experimental work to date, performance of an example embodiment ofpattern recognition model 1002 as a CNN has been evaluated. Using a CNNsimilar to pattern recognition model 1002 described herein, training andvalidation were performed. FIG. 50 shows plots of model loss and modelaccuracy on training and validation datasets over training epochs.

A dataset was generated of 13,324 images of cross-section regions ofinterest (ROIs).

CNN training was performed on a training dataset of 10754 regions ofinterest (80% of the total available dataset) and validation wasperformed with the remaining 2570 regions of interest (20% of the totalavailable dataset).

As seen in FIG. 50, the model has comparable performance on bothtraining and validation datasets, as the plots converge.

Image Segmentation

Classifier 1000 is configured to perform segmentation of unprocessedhigh-resolution DNIRA images to generate a segmented image. A segmentedimage may be a masked image, shown for example in a masked image 1404shown in FIG. 51.

Image segmentation refers to a process of identifying a set of pixels inan image belonging to an object or feature of interest. In particular,in some embodiments classifier 1000 segments retinal images andidentifies areas where retinal tissue changes have occurred.

By way of example, regions of hypofluorescent/black DNIRA can besegmented, as shown in FIG. 51, illustrating an input 1402, a DNIRA enface ocular image, to classifier 1000, and a masked image 1404 ofdefined shapes generated by classifier 1000.

In some embodiments, features generated by classifier 1000 includedefined areas of hypofluorescence (that identify regions of absentfluorescence signal), or areas of hyperfluorescence, as illustrated inmasked image 1404.

Input images may be evaluated by classifier 1000 at a single timepointand regions of profoundly hypofluorescent/black DNIRA signal identifiedand quantified.

Input 1402 illustrates a raw image of a patient with late dry AMD, withregions of hypo/black DNIRA.

Output 1404 illustrates an automatically segmented image, or mask,derived using classifier 1000.

Images may be obtained at a single timepoint, or over multipletimepoints for processing by classifier 1000. Regions of profoundlyhypofluorescent/black DNIRA signal may be identified and quantified, andthe difference, or delta, identified. FIG. 52 shows an example sequenceof masked images from four timepoints. As shown, the mask images may begenerated for multi-modal image input, e.g., FAF and DNIRA images.

In an embodiment of pattern recognition model 1002, U-Net convolutionalnetworks may be utilized to segment 2D FAF and DNIRA images and extractareas where signal loss has occurred.

The U-Net convolutional network works by using convolutional layers toproduce a low-dimensionality representation of the image. The U-Netconvolutional network then creates a new representation of the image bygoing through de-convolutional layers. The de-convolutional networksuppresses undesired features in the image.

In some embodiments, implementation of the U-Net architecture uses batchnormalization. Batch normalization refers to normalizing the output ofdifferent layers in neural networks. This may improve information flowfrom one layer to the next, which may allows the algorithm to convergefaster.

Image segmentation may be performed using a classical machine learningapproach, or a deep learning based approach.

In an example classical machine learning approach:

-   -   1. Construct a training set by selecting regions with        hypofluorescent DNIRA (class 1) and regions where        non-hypofluorescent DNIRA (class2, normal DNIRA signal and/or        hyperfluorescent DNIRA).    -   2. Extract a set of predetermined features (mean, variance,        entropy, texture features, wavelet features) for both regions.    -   3. Train an algorithm to distinguish between class1 and class2.        These algorithms are generally based on decision trees such as        random forest trees.    -   4. Define window size that slides across the image.    -   5. At each window step extract same set of features as in step        2.    -   6. Use trained algorithm in step 3 to decide whether this region        is hyperfluorescent DNIRA or not.    -   7. Decision map is now a segmentation map used to extract        hypofluorescent regions.

In an example deep learning based approach:

-   -   1. Use multi-level threshold create binary maps of DNIRA. At        each level identify regions where hyperfluorescent DNIRA outline        has been identified. Combine these regions and set create a        binary mask where 1 indicates pixel belongs to hypofluorescent        region (class 1) and 0 indicates that pixel belongs to class 2.    -   2. Save images and masks in separate folders. Both the image and        the mask should have the same name identifier.    -   3. Build a large data bank containing many examples of manually        generated masks which will be used to train deep learning        algorithms.    -   4. Use deep learning based segmentation algorithm using        generated data bank in step 3.        (https://ieeexplore.ieee.org/document/7749007).    -   5. Store weights for trained algorithm on disk.    -   6. Load algorithm and training weights and pass new image        through the algorithm.    -   7. Algorithm outputs probability map for the likelihood that a        certain pixel belongs to class A or B. to generate a binary mask        a probability threshold is applied (in our case we use a 50% as        the threshold).

In some embodiments, image segmentation may be performed using imageprocessing algorithms such as edge detection or principal componentanalysis, alone or in combination with techniques described herein.

In some embodiments, training makes use of boot-strapping to improve thequality of segmentation algorithms. In boot-strapping masks can begenerated using the classical machine learning approach which are thenmanually examined and improved to be used as part of the training bankfor deep learning based approaches.

Feature Classification

The features classified by classifier 1000 may be phenotypes of a userassociated with the plurality of ocular images.

In some embodiments, the features may distinguish between differentphenotypes and biomarkers.

In some embodiments, the features may be correlated with stage or gradevariants of blinding eye disease including Age Related MacularDegeneration (AMD), monogenic eye disease, inherited eye disease andinflammatory eye disease.

In some embodiments, the features may be correlated with stage or gradevariants of central nervous system (brain) disease, such as dementia andAlzheimer's disease.

In some embodiments, the features may be correlated with other measuresof disease or health.

In some embodiments, the features may be correlated with biomarkers, byway of non-limiting example such as those for diagnosis, prognosis,monitoring, and those applied to determine treatment efficacy or safety.

The output of classifier 1000 may be, by way of non-limiting example,categorical, continuous, binary, or descriptive.

Registration

In some embodiments, ocular images may be registered to a commoncoordinate system before classification. In some embodiments, ocularimages may be registered following classification.

In some embodiments, registration may be cross-modal fusion of theplurality of ocular images to a common coordinate system.

In some embodiments, registration may be done using elastic ornon-elastic transformation.

As illustrated in FIG. 53, in some embodiments, image registration isperformed by first scaling the images, such as unprocessedhigh-resolution DNIRA images, and applying a Gaussian filter. ScaleInvariant Feature Transform (SIFT, included in OpenCV) is used to findfeatures across multiple images. The main features of interest in anexample may be blood vessels. The corresponding feature pairs betweentwo images are determined using brute force algorithm (BF) followed byrandom sample consensus (RANSAC) to calculate an affine transformationmap. The inverse of the affine matrix is applied to the second image toregister it to the first image.

Multi-modal registration may be used to align images of the samemodality, or between DNIRA and images of other modalities.

Segmentation Map

Returning to FIG. 49, in some embodiments, classifier 1000 may performsegmentation 1040 based on input cross-section images. At block 1042,cross-section segmentation map is generated corresponding to an en faceregion of an eye, each segment of the cross-section segmentation mapcorresponding to a cross-section image at that region of the eye.

At block 1044, each segment of the cross-section segmentation map isclassified as a phenotype such as a tissue state, for example, one ofpseudodrusen, normal, drusen, retinal pigment epithelium detachment,geographical atrophy, macular atrophy or neovascularization based atleast in part on classification of the cross-section image correspondingto that segment using pattern recognition model 1002.

FIG. 54 illustrates an example classified cross-section segmentation map1502, with the classifications of tissue state phenotypes identified inthe legend as psuedodrusen, normal, drusen or GA (geographical atrophy).

Steps of segmentation 1040 may be performed to generate a cross-sectionsegmentation map for each of the multiple time points. A time seriesdata model may then be generated based on the cross-section segmentationmap at each of the multiple time points.

As shown in FIG. 55, in some embodiments, a classified cross-sectionsegmentation map 1504 may be registered to a common coordinate systemwith an en face FAF image 1506, illustrated by registered image 1508.

Conveniently, cross-section segment information on a eye, such as atissue state, may be mapped to the corresponding en face region tovisualize cross-section information on an en face image, providingfurther insights on the health of an eye.

Time Series

FIG. 56 illustrates classified cross-section segmentation maps 1602A,1602B, 1602C, 1602D and 1602E, each associated with a corresponding timepoint. Classified cross-section segmentation maps 1602A-1602B aregenerated based on cross-sections of ocular images, illustrating aprogression or change in eye tissue over time. Classifications of tissuestate phenotypes identified in the legend illustrated in FIG. 56 aspsuedodrusen, normal, drusen or GA (geographical atrophy). In someembodiments, a time series of segmentation maps such as those shown inFIG. 56 may be used to train a predictor, for example, a time seriesmodel, to predict whether tissue loss has occurred or will occur withina given time period. Different time scales may be used for trainingexamples, for example, across one or more of the maps 1602A-1602E. Thus,the presence of certain features (such as a region of particular size,shape and tissue type) may be used to predict tissue loss, for example,future rates of geographical atrophy expansion, based on such a timeseries model.

In some embodiments, the time series data model is based at least inpart on identified changes in the cross-section segmentation maps overtime.

In some embodiments, the time series data model visualizes diseaseprogression.

In some embodiments, the time series data model is based at least inpart on elapsed time between the multiple time points.

Multi-Sessional Image Registration

Retinal imaging of a patient population may be used to keep track ofretinal tissue loss in time, and patients come back for multiple imagingsessions. Temporal analysis of the rate of tissue loss in time is aquantifiable observation and key requirement when performing medicaltrials. Thus, image registration may also be applied for themultisession images for each individual patient.

A registration pipeline used in some embodiments to perform multimodalimage acquisition for multiple sessions includes registering en faceimages to the cross-sectional images. Cross-sectional OCT images may beacquired using the Infrared (IR) wavelength and the En face OCTregistered to the IR confocal image. The reference IR image may be usedto register the other modalities, thus allowing multi-modal registrationof OCT to other 2D images.

FIG. 57 illustrates an example of en face OCT using IR image referenceto determine position of cross-section. This IR reference is used toregister other modalities to the OCT.

FIG. 58 illustrates multi-modal image registration for multiplesessions. The IR image used by the OCT in the first session (S1) is usedto register images in all the sessions (S1, S2 and S3).

As shown by way of example in FIG. 58, the confocal IR image in thefirst session (S1) is used as the reference to register all the otherimages. FAF in S1 is registered to IR in S1. DNIRA in S1 is registeredto FAF in S1. For imaging sessions beyond S1, each image is registeredto its corresponding modality. For example, DNIRA in S2 is registered toDNIRA in S1. DNIRA in S3 is registered to the “now registered” DNIRA inS2. This registration path may result in a high level of consistencybetween extracted features from one session to the other. Conveniently,this may provide a better overall registration performance.

In another example, OCT scans may be performed in the near IR channel,as shown in FIG. 59 and the pipeline may use the DNIRA image as areference, as shown in FIG. 60.

FIG. 60 illustrates multi-modal image registration pipeline for multiplesessions. The DNIRA image used by the Near IR OCT in the first session(S1) is used to register images in all the sessions (S1, S2 and S3).

Detector

The computer systems described herein may also be adapted to provide acomputer system that processes ophthalmic image data to automaticallydetect the presence of image-based phenotypes, and disease features, eyestructures, or other features of interest. Each of such special purposecomputer systems may be referred to as a “detector” herein.

In some embodiments, a detector may be configured to detect presence ofone or more of the features shown in FIGS. 21A-21E, namely, according topre-defined descriptive labels “loose weave”, “tight weave”, “leopardspots”, “grey smudge”, “fingerling potatoes” or the like.

As described further herein, features may include hypofluorescentpatterns such as: a region of repeating hypofluroescent shapes, forexample, in a “loose weave” pattern; a region of concentrated repeatinghypofluorescent shapes, for example, in a “tight weave” pattern;patterns of generally elliptical, oval or oblong shaped hypofluorescentregions, for example, hypofluroescent shapes resembling “fingerlingpotatoes”; patterns of generally round or rosette shaped hypofluorescentregions, for example, hypofluorescent shapes resembling “leopard spots”;and regions of intermediate hypofluroescence (IHoF) or lowhypofluorescence (LHoF), for example, resembling a “grey smudge”.

In some embodiments, a detector may be configured to detect the presencephagocytic immune cells such as macrophages.

Optionally, the detector may generate one or more descriptors of thecharacteristics of the detected phenotype or feature, such as location,size, quantity, color.

FIG. 46B depicts a detector 1100 exemplary of an embodiment. Asdepicted, detector 1100 may receive image data and non-image data asinputs and processes these data to produce a detection indicator,namely, a digital indicator of whether a particular feature or phenotypewas detected, as an output.

The detection indicator may be, by way of non-limiting example,continuous, binary, or descriptive.

The image data input to detector 1100 may include some or all of thetypes of data described in association with classifier 1000.

The non-image data input to detector 1100 may include some or all of thetypes of data described in association with classifier 1000.

In some embodiments, detector 1100 may be configured to performdetection using solely image data input.

The operation of detector 1100 to detect a phenotype is described withreference to the flowchart of FIG. 61.

At block 1102, input image data, for example, ocular imagescorresponding to a plurality of imaging modalities, is received.

The imaging modalities may include, for example, delayed near-infraredanalysis (DNIRA), infra-red reflectance (IR), confocal scanning laserophthalmoscopy (cSLO), fundus autofluorescence (FAF), color fundusphotography (CFP), optical coherence tomography (OCT), OCT-angiography,fluorescence lifetime imaging (FLI), multispectral detection, andpolarized fluorescence imaging.

At block 1104, the ocular images are registered to a common coordinatesystem, for example, using registration techniques as described herein.

At block 1106, features of each of the ocular images are classified,using a pattern recognition model.

In some embodiments, the pattern recognition model may be patternrecognition model 1102, as described herein. For example, the patternrecognition model may be a convolutional neural network built based ontraining data corresponding to a plurality of ocular images and featureextraction and feature selection is performed to generate features fromthe training data.

In some embodiments, the feature extraction generates a greyscale imageof defined shapes. The defined shapes may include at least one ofpatterns labelled as “leopard spots”, “loose weave”, “grey smudge”, and“fingerling potatoes”, as described further herein.

For example, defined shapes may include hypofluorescent patterns suchas: a region of repeating hypofluroescent shapes, for example, in a“loose weave” pattern; a region of concentrated repeatinghypofluorescent shapes, for example, in a “tight weave” pattern;patterns of generally elliptical, oval or oblong shaped hypofluorescentregions, for example, hypofluroescent shapes resembling “fingerlingpotatoes”; patterns of generally round or rosette shaped hypofluorescentregions, for example, hypofluorescent shapes resembling “leopard spots”;and regions of intermediate hypofluroescence (IHoF) or lowhypofluorescence (LHoF), for example, resembling a “grey smudge”.

At block 1108, features of the image are identified as phenotypes.

In some embodiments, detector 1100 may further correlate one or more ofthe defined shapes with the presence of phagocytic immune cells such asmacrophages.

In some embodiments, detector 1100 may further generate one or moredescriptors of characteristics of the identified phenotype, such aslocation, size, quantity and colour.

Predictor

The computer systems described herein may also be adapted to provide acomputer system that processes ophthalmic image data to automaticallypredict future changes in a patient, e.g., regions of diseaseprogression, rate of disease progress. Each of such special purposecomputer systems may be referred to as a “predictor” herein.

In some embodiments, a predictor may be configured to predict an area oftissue loss, or a rate of tissue loss. In some embodiments, a predictormay be configured to predict progression from early to late dry AMD. Insome embodiments, a predictor may be configured to predict the responseof a patient or group of patients to an intervention (e.g., a therapy).In some embodiments, a predictor may be configured to predict futureonset of new GA. In some embodiments, a predictor may be configured topredict neovascularization.

In some embodiments, a predictor may compute one or more metricsreflective of a confidence level of a particular prediction.

FIG. 46C depicts a predictor 1200 exemplary of an embodiment. Asdepicted, predictor 1200 may receive image data and non-image data asinputs and processes these data to produce a prediction indicator,namely, a digital indicator of a computed prediction, as an output.

The image data input to predictor 1200 may include some or all of thetypes of data described in association with classifier 1000.

The non-image data input to predictor 1200 may include some or all ofthe types of data described in association with classifier 1000.

The image data input to predictor 1200 may include image data for imagesobtained at a single time point (a single session) or over multiple timepoints (multiple sessions). Predictor 1200 may be trained using imagedata for multiple time points to learn trends in features or phenotypes.For example, FIG. 62 illustrates images taken of a particular subject'seye over five sessions, showing an increase in hypofluorescent/black inthe DNIRA image over that time.

In some embodiments, predictor 1200 may generate a prediction indicatorreflective of a predicted change over time, either for an individualsubject or a group of subjects. The change may be expressed in a varietyof units, for example, the change in total area, the changes in thesquare root of the total area, the change in focality, the change in thefocality index, or the like. Other examples include Circularity,Equivalent Diameter, Solidty, Eccentricity, Extent, Aspect Ratio.Spatial moment analysis may also be applied to these patterns, known asHu invariants. The prediction indicator may be, by way of non-limitingexample, categorical, continuous, binary, or descriptive.

The operation of predictor 1200 to predict tissue loss is described withreference to the flowchart of FIG. 63.

At block 1202, input image data, for example, ocular imagescorresponding to a plurality of imaging modalities, is received.

The imaging modalities may include, for example, delayed near-infraredanalysis (DNIRA), infra-red reflectance (IR), confocal scanning laserophthalmoscopy (cSLO), fundus autofluorescence (FAF), color fundusphotography (CFP), optical coherence tomography (OCT), andOCT-angiography.

The imaging modalities may include cross-section images and en faceimages.

At block 1204, the ocular images are registered to a common coordinatesystem, for example, using registration techniques as described herein.

At block 1206, features of each of the ocular images are classified,using a pattern recognition model.

In some embodiments, the pattern recognition model may be patternrecognition model 1002, as described herein. For example, the patternrecognition model may be a convolutional neural network built based ontraining data corresponding to a plurality of ocular images and featureextraction and feature selection is performed to generate features fromthe training data.

At block 1208, tissue loss may be predicted based at least in part onthe features. In some embodiments, choroidal neovascularization (CNV)may be predicted, based at least in part on the features.

In some embodiments, the features selected comprise phenotypes of a userassociated with the plurality of ocular images.

In some embodiments, tissue loss prediction may include identifying thephenotypes as risk factors by correlating the phenotypes with a rate oftissue loss over time.

In some embodiments, the predicting tissue loss is based on time seriesforecasting to predict tissue loss based on a time series data model,such as time series data model as generated by classifier 1000 asdescribed herein.

For example, the time series data model may be generated based onmultiple cross-section segmentation maps generated for each of multipletime points and corresponding cross-section images, each of thecross-section segmentation maps corresponding to an en face region of aneye, and each segment of the cross-section segmentation mapcorresponding to a cross-section image at that region of the eyeclassified as a phenotype of one of pseudodrusen, normal, drusen,retinal pigment epithelium detachment, geographical atrophy, macularatrophy or neovascularization based at least in part on classificationof the cross-section image corresponding to that segment using aconvolutional neural network.

In some embodiments, predicting tissue loss is based at least in part onnon-image based biomarker data. Non-image based biomarker data mayinclude characteristics of a user associated with the plurality ofocular images, the characteristics including at least one of age,genetics, sex, smoker status, and diet.

Predicting tissue loss may include one or more of predicting a rate oftissue loss, predicting whether tissue loss has previously occurred, andpredicting whether tissue loss will occur in the future.

In some embodiments, predicting tissue loss comprises predicting regionsof disease progression and rate of disease progress, such as progressionfrom early to late dry Age Related Macular Degeneration (AMD).

In some embodiments, predictor 1200 may predict choroidalneovascularization (CNV) which indicates that the patient has convertedfrom dry to wet AMD. In some embodiments, predicting neovascularizationcomprises predicting a onset of neovascularizaton. In some embodiments,predicting neovascularization comprises predicting whetherneovascularization has previously occurred. In some embodiments,predicting neovascularization comprises predicting whetherneovascularization will occur in the future.

In some embodiments, two or more classifiers 1000, detectors 1100, andpredictors 1200 operate in concert to produce a category indicator, adetection indicator, or a prediction indicator. In one particularexample, a detection indicator indicating the presence of a particularfeature or phenotype (e.g., a macrophage) may be included withinnon-image data inputted to a predictor 1200. In this way, for example, aprediction of disease progression may be made based on the detection ofa macrophage. In another particular example, a detection indicatorindicating the presence of a particular feature or phenotype may beincluded within non-image data inputted to a classifier 1100. In anotherparticular example, a classifier indicator may be included withinnon-image data inputted to a detector 1000. In another particularexample, a classifier indicator may be included within non-image datainputted to a predictor 1200. In another particular example, aprediction indicator may be included within non-image data inputted to aclassifier 1000. In another particular example, a prediction indicatormay be included within non-image data inputted to a detector 1100.

Other combinations of classifiers 1000, detectors 1100, and predictors1200 will be readily apparent to one of ordinary skill in the art inview of the present disclosure.

In some embodiments, the output of a particular classifier 1000,detector 1100, and predictor 1200 may be used to train that particularclassifier 1000, detector 1100, or predictor 1200. Such training may besupervised, partially supervised, or unsupervised. In some embodiments,the output of one or more classifiers 1000, detectors 1100, andpredictors 1200 may be used to train others of classifiers 1000,detectors 1100, and predictors 1200.

Some embodiments of the classifiers 1000, detectors 1100, and predictors1200 disclosed herein may have therapeutic value. In one exampleembodiment, a classifier 1000, detector 1100, or predictor 1200 usingimage data including DNIRA image data may be used in a method fordetermining a potential treatment or intervention will be useful intreating Age Related Macular Degeneration or other diseases as describedherein. For example, a classifier 1000, detector 1100, or predictor 1200may generate or use one or more features or phenotypes corresponding toa biomarker in a method for determining a potential treatment orintervention will be useful in treating Age Related Macular Degenerationor other diseases. In one specific example, the treatment orintervention may be one that targets macrophages.

In another example embodiment, a classifier 1000, detector 1100, orpredictor 1200 using image data including DNIRA image data may be usedin a method for determining that compound may be useful in treating AgeRelated Macular Degeneration or other diseases, the method comprisingassaying the eye with DNIRA-based feature or phenotypes, madedistinguishable using the classifier 1000, and administering atherapeutically effective amount of the compound to the patient if aDNIRA-based feature or phenotype is present. For example, a classifier1000, detector 1100, or predictor 1200 may generate or use one or morefeatures or phenotypes corresponding to a biomarker in for determiningthat compound may be useful in treating Age Related Macular Degenerationor other diseases, the method comprising assaying the eye withDNIRA-based feature or phenotypes, made distinguishable using theclassifier 1000, detector 1100, or predictor 1200, and administering atherapeutically effective amount of the compound to the patient if aDNIRA-based feature or phenotype is present. In one specific example,the compound may be particular to a compound that targets macrophages.

FIG. 64 illustrates an embodiment of predictor 1200 to predict diseaseprogression based on feature extraction (for example, performed byclassifier 1000) and cross-modal analytics using CNNs that arepre-trained on a plurality of image modalities, and correlated (orregistered) with OCT images, to predict and follow changes to tissue.

FIG. 64 illustrates an example of a technique for detecting a featuresuch as a soft drusen (in a colour picture where they are identified)then performing cross-modal analysis and training of a CNN to confirmusing the OCT that it is in fact a class drusen or drusenoid RPEdetachment, and from that information, feed this into the OCTsegmentation maps, as described herein.

Multimodal images and divide them into small regions of interest (ROIs).Each modality may contain different information about the tissueanomaly. Predictor 1200 combines this information and determines andoutputs if there is no change, GA formation/expansion has occurredwithin a given time frame, and/or CNV.

The classifier may utilize different features from different imagingmodalities. Each CNN may be trained a simpler task the convolutionallayers used as input for the prediction stage.

For the case of OCT, a first CNN model, CNN1, is trained to determinewhether tissue anomaly is normal, pseudodrusen, drusen or GA. For FAF, asecond CNN model, CNN2, is trained to determine whether there is GA orno GA in the ROI. For DNIRA, a third CNN model, CNN3, is trained todetermine whether this is normal tissue or hypofluorescent.

The pre-trained convolutional layers are then used to extract featuresfrom multimodal ROI images. Those features are combined (feature levelfusion) along with other features about the patient health status (forexample, age and smoking states). The predictor is also fed informationabout the time difference between the two session.

Predictor 1200 then outputs a prediction of no change, new GAformation/expansion, and/or CNV.

As noted above, predictor 1200, using image data including DNIRA imagedata, may be used in determining that a potential treatment orintervention will be useful in treating diseases. In some embodiments,this may include evaluating or determining whether an agent is effectivefor the treatment of an ocular disorder as described herein.

In some embodiments, the agent is an immunomodulatory agent (optionallyselected from a Monocyte Chemoattractant Protein (MCP)-modulating agent,inclusive of MCP-1, MCP-2, and MCP-3-modulating agents, including acompound of Formula I (2-((1-benzylindazol-3-yl) methoxy)-2-methylpropionic acid), such as bindarit (sometimes referred to herein asTMi-018), methotrexate, PPAR gamma modulator, migration inhibitoryfactor (MIF) inhibitor, and chemokine receptor 2 (CCR2) inhibitor (e.g.,Maraviroc, cenicriviroc, CD192, CCX872, CCX140,2-((Isopropylaminocarbonyl)amino)-N-(2-((cis-2-((4-(memylthio)benzoyl)amino)cyclohexyl)amino)-2-oxoethyl)-5-(trifluoromethyl)-benzamide,vicriviroc, SCH351125, TAK779, Teijin, RS-504393, compound 2, compound14, or compound 19 (Plos ONE 7(3): e32864)).

In some embodiments, the agent is a complement factor. In someembodiments, the agent is an anti-factor D antibody (e.g. lampalizumab(Genentech)), an anti-amyloid (anti-Aβ) antibody (e.g. GSK933776 (GSK)),a corticosteroid (e.g. fluocinolone acetonide), MC-1101 (MacuCLEAR), aCD34+ stem cell therapy, an anti-VEGF antibody (e.g. Ranibizumab),brimonidine (Alphagan), an anti-C5 complement antibody (e.g. LFG316(Novartis), doxycycline (ORACEA), emixustat hydrochloride, sirolimus(RAPAMUNE), and glatiramer acetate (COPAXONE).

In some embodiments, the agent is a nucleoside reverse transcriptaseinhibitor (NRTIs), by way of non-limiting example zidovudine,didanosine, zalcitabine, stavudine, lamivudine, abacavir, emtricitabine,and entecavir. In some embodiments, the agent is acyclovir.

The agent may also be, in various embodiments, an anti-vascularendothelial growth factor (VEGF) agent (e.g., Ranibizumb (LUCENTIS),Bevizumab (AVASTIN) or Aflibercept (EYLEA)), an angiotensin-convertingenzyme (ACE) inhibitor, a peroxisome proliferator-activated receptor(PPAR)-gamma agonist (e.g., rosiglitazone (AVANDIA), pioglitazone(ACTOS), troglitazone (REZULIN), netoglitazone, rivoglitazone,ciglitazone, rhodanine), a renin inhibitor, a steroid, and an agent thatmodulates autophagy. In some embodiments, the agent is a modulator ofthe complement cascade (e.g. a modulator of C3, C5, complement factor D,or complement factor B).

In still another embodiment, an agent is a modulator of macrophagepolarization. Illustrative modulators of macrophage polarization includeperoxisome proliferator activated receptor gamma (PPAR-g) modulators,including, for example, agonists, partial agonists, antagonists orcombined PPAR-gamma/alpha agonists.

In some embodiments, the PPAR gamma modulator is a full agonist or apartial agonist. In some embodiments, the PPAR gamma modulator is amember of the drug class of thiazolidinediones (TZDs, or glitazones). Byway of non-limiting example, the PPAR gamma modulator may be one or moreof rosiglitazone (AVANDIA), pioglitazone (ACTOS), troglitazone(REZULIN), netoglitazone, rivoglitazone, ciglitazone, rhodanine. In someembodiments, the PPAR gamma modulator is one or more of irbesartan andtelmesartan. In some embodiments, the PPAR gamma modulator is anonsteroidal anti-inflammatory drugs (NSAID, such as, for example,ibuprofen) and indoles. Known inhibitors include the experimental agentGW-9662. Further examples of PPAR gamma modulators are described in WIPOPublication Nos. WO/1999/063983, WO/2001/000579, Nat Rev Immunol. 2011Oct. 25; 11(11):750-61, or agents identified using the methods ofWO/2002/068386, the contents of which are hereby incorporated byreference in their entireties.

In some embodiments, the PPAR gamma modulator is a “dual,” or“balanced,” or “pan” PPAR modulator. In some embodiments, the PPAR gammamodulator is a glitazar, which bind two or more PPAR isoforms, e.g.,muraglitazar (Pargluva) and tesaglitazar (Galida) and aleglitazar.

In another embodiment, an agent is semapimod (CNI-1493) as described inBianchi, et al. (March 1995). Molecular Medicine (Cambridge, Mass.) 1(3): 254-266, the contents of which are hereby incorporated by referencein their entireties.

In still another embodiment, an agent is a migration inhibitory factor(MIF) inhibitor. Illustrative MIF inhibitors are described in WIPOPublication Nos. WO 2003/104203, WO 2007/070961, WO 2009/117706 and U.S.Pat. Nos. 7,732,146 and 7,632,505, and 7,294,753 7,294,753 the contentsof which are hereby incorporated by reference in their entireties. Insome embodiments, the MIF inhibitor is(S,R)-3-(4-hydroxyphenyl)-4,5-dihydro-5-isoxazole acetic acid methylester (ISO-1), isoxazoline, p425 (J. Biol. Chem., 287, 30653-30663),epoxyazadiradione, or vitamin E.

In still another embodiment, an agent is a chemokine receptor 2 (CCR2)inhibitor as described in, for example, U.S. Patent and PatentPublication Nos.: U.S. Pat. Nos. 7,799,824, 8,067,415, US 2007/0197590,US 2006/0069123, US 2006/0058289, and US 2007/0037794, the contents ofwhich are hereby incorporated by reference in their entireties. In someembodiments, the CCR2) inhibitor is Maraviroc, cenicriviroc, CD192,CCX872, CCX140,2-((Isopropylaminocarbonyl)amino)-N-(2-((cis-2-((4-(memylthio)benzoyl)amino)cyclohexyl)amino)-2-oxoethyl)-5-(trifluoromethyl)-benzamide,vicriviroc, SCH351125, TAK779, Teijin, RS-504393, compound 2, compound14, or compound 19 (Plos ONE 7(3): e32864).

In various embodiments, an agent is one or more of CKR-2B, a2-thioimidazole, CCR2 Antagonist (CAS 445479-97-0), and CCX140.

In various embodiments an agent is an anti-VEGF agent. Non-limitingexamples of anti-VEGF agents useful in the present methods includeranibizumab, bevacizumab, aflibercept, KH902 VEGF receptor-Fc, fusionprotein, 2C3 antibody, ORA102, pegaptanib, bevasiranib, SIRNA-027,decursin, decursinol, picropodophyllin, guggulsterone, PLG1O1,eicosanoid LXA4, PTK787, pazopanib, axitinib, CDDO-Me, CDDO-Imm,shikonin, beta-, hydroxyisovalerylshikonin, ganglioside GM3, DC101antibody, Mab25 antibody, Mab73 antibody, 4A5 antibody, 4E10 antibody,5F12 antibody, VA01 antibody, BL2 antibody, VEGF-related protein,sFLTO1, sFLT02, Peptide B3, TG100801, sorafenib, G6-31 antibody, afusion antibody and an antibody that binds to an epitope of VEGF.Additional non-limiting examples of anti-VEGF agents useful in thepresent methods include a substance that specifically binds to one ormore of a human vascular endothelial growth factor-A (VEGF-A), humanvascular endothelial growth factor-B (VEGF-B), human vascularendothelial growth factor-C (VEGF-C), human vascular endothelial growthfactor-D (VEGF-D) and human vascular endothelial growth, factor-E(VEGF-E), and an antibody that binds, to an epitope of VEGF.

In one embodiment, the anti-VEGF agent is the antibody ranibizumab or apharmaceutically acceptable salt thereof. Ranibizumab is commerciallyavailable under the trademark LUCENTIS. In another embodiment, theanti-VEGF agent is the antibody bevacizumab or a pharmaceuticallyacceptable salt thereof. Bevacizumab is commercially available under thetrademark AVASTIN. In another embodiment, the anti-VEGF agent isaflibercept or a pharmaceutically acceptable salt thereof. Afliberceptis commercially available under the trademark EYLEA. In one embodiment,the anti-VEGF agent is pegaptanib or a pharmaceutically acceptable saltthereof. Pegaptinib is commercially available under the trademarkMACUGEN. In another embodiment, the anti-VEGF agent is an antibody or anantibody fragment that binds to an epitope of VEGF, such as an epitopeof VEGF-A, VEGF-B, VEGF-C, VEGF-D, or VEGF-E. In some embodiments, theVEGF antagonist binds to an epitope of VEGF such that binding of VEGFand VEGFR are inhibited. In one embodiment, the epitope encompasses acomponent of the three dimensional structure of VEGF that is displayed,such that the epitope is exposed on the surface of the folded VEGFmolecule. In one embodiment, the epitope is a linear amino acid sequencefrom VEGF.

In various embodiments, an agent is a renin angiotensin system (RAS)inhibitor. In some embodiments, the renin angiotensin system (RAS)inhibitor is one or more of an angiotensin-converting enzyme (ACE)inhibitor, an angiotensin-receptor blocker, and a renin inhibitor.

Non-limiting examples of angiotensin-converting enzyme (ACE) inhibitorswhich are useful in the present embodiments include, but are not limitedto: alacepril, alatriopril, altiopril calcium, ancovenin, benazepril,benazepril hydrochloride, benazeprilat, benzazepril, benzoylcaptopril,captopril, captoprilcysteine, captoprilglutathione, ceranapril,ceranopril, ceronapril, cilazapril, cilazaprilat, converstatin,delapril, delaprildiacid, enalapril, enalaprilat, enalkiren, enapril,epicaptopril, foroxymithine, fosfenopril, fosenopril, fosenopril sodium,fosinopril, fosinopril sodium, fosinoprilat, fosinoprilic acid,glycopril, hemorphin-4, idapril, imidapril, indolapril, indolaprilat,libenzapril, lisinopril, lyciumin A, lyciumin B, mixanpril, moexipril,moexiprilat, moveltipril, muracein A, muracein B, muracein C, pentopril,perindopril, perindoprilat, pivalopril, pivopril, quinapril, quinaprilhydrochloride, quinaprilat, ramipril, ramiprilat, spirapril, spiraprilhydrochloride, spiraprilat, spiropril, spirapril hydrochloride,temocapril, temocapril hydrochloride, teprotide, trandolapril,trandolaprilat, utibapril, zabicipril, zabiciprilat, zofenopril,zofenoprilat, pharmaceutically acceptable salts thereof, and mixturesthereof.

Non-limiting examples of angiotensin-receptor blockers which are usefulin the present embodiments include, but are not limited to: irbesartan(U.S. Pat. No. 5,270,317, hereby incorporated by reference in itsentirety), candesartan (U.S. Pat. Nos. 5,196,444 and 5,705,517 herebyincorporated by reference in their entirety), valsartan (U.S. Pat. No.5,399,578, hereby incorporated by reference in its entirety), andlosartan (U.S. Pat. No. 5,138,069, hereby incorporated by reference inits entirety).

Non-limiting examples of renin inhibitors which are useful in thepresent embodiments include, but are not limited to: aliskiren,ditekiren, enalkiren, remikiren, terlakiren, ciprokiren and zankiren,pharmaceutically acceptable salts thereof, and mixtures thereof.

In various embodiments an agent is a steroid. In some embodiments, asteroid is a compound belonging to or related to the followingillustrative families of compounds: corticosteroids, mmeralicosteroids,and sex steroids (including, for example, potentially androgenic orestrogenic or anti-andogenic and anti-estrogenic molecules). Includedamongst these are, by way of non-limiting example, prednisone,prednisolone, methyl-prednisolone, triamcinolone, fluocinolone,aldosterone, spironolactone, danazol (otherwise known as OPTINA), andothers.

In various embodiments an agent modulates autophagy, microautophagy,mitophagy or other forms of autophagy. In some embodiments, thecandidate drug and/or compound is one or more of sirolimus, tacrolimis,rapamycin, everolimus, bafilomycin, chloroquine, hydroxychloroquine,spautin-1, metformin, perifosine, resveratrol, trichostatin, valproicacide, Z-VAD-FMK, or others known to those in the art. Without wishingto be bound by theory, agent that modulates autophagy, microautophagy,mitophagy or other forms of autophagy may alter the recycling ofintra-cellular components, for example, but not limited to, cellularorganelles, mitochondria, endoplasmic reticulum, lipid or others.Without further wishing to be bound by theory, this agent may or may notact through microtubule-associated protein 1A/1B-light chain 3 (LC3).

The systems and methods disclosed herein are further described by thefollowing non-limiting examples.

EXAMPLES Example 1: DNIRA to Identify, Quantify and Follow Regions ofProfoundly Hypofluorescent/Black Signal

As analysis of DNIRA has not been previously reported, the inventorsbelieve all observations to be novel. DNIRA can be used to identifyregions of profoundly hypofluorescent/black signal. These may or may notcorrespond with regions or distinguishing features identified by otherimaging modalities. Unlike the profoundly hypofluorescent/black signalof FAF that represents absent RPE/photoreceptors and therefore remainsthe same in size or expands over time, dark regions of DNIRA are dynamicand can enlarge, decrease or remain the same. As such, softwarealgorithms have been developed to identify and quantify the extent ofabsent DNIRA signal, and are predicated, without limitation, on softwarefor registration, alignment, contrast adjustment, sharpening andsegmentation. FIG. 10 is a graphical illustration of DNIRA imageanalysis

Subject consent is used for all clinical research protocols are ResearchEthics Board (REB) approved and subjects willing and able to consent areserially enrolled.

Baseline in vivo imaging: baseline AMDI images are acquired using acommercially available confocal scanning laser ophthalmoscope (cSLO)(e.g. Heidelberg Retinal Angiography, HRA-2, Heidelberg Engineering,Germany). Images are obtained prior to systemic injection using cSLO,and those with significant optical opacity are eliminated from furtherimaging (other than as clinically indicated). Amongst subjects withsuitable optical clarity, additional images are also obtained in thered-free, FAF (488/500 nm excitation/emission), IR reflectance channel(830 nm) and ICG fluorescence channels (795/810 nm excitation/emission),along with optical coherence tomography (OCT) and color fundusphotography. Within a suitably short time-frame, for example between 1and 28 days, subjects undergo ICG injection that may or may not includeangiography.

ICG injection: ICG dye (Cardiogreen, Sigma) is freshly prepared prior toexperimentation to a final stock concentration of 5.0 mg/ml in sterilewater. A fine gauge catheter is inserted intravenously, and ICG dye isinfused. If angiography is performed, images are acquired prior toinjection (baseline), during dye circulation, and at various intervalsthereafter typically out to 40 minutes or 2 hours. Alternatively, ICGcan be provided by oral administration, for example, by way of a acapsule, tablet, solution, or suspension.

AMD imaging: Within a suitable time-frame, for example 1 to 5 days (butpotentially 24 hours to 120 days after ICG), subjects return for DNIRAimages obtained with ICG excitation/emission filters or laser systems inplace (795/810 nm) but without further injection of dye. Multiple imagesare acquired of each eye including those centered on the fovea and theoptic nerve head. Images are also taken in four quadrants of the macula,thereby permitting visualization of a larger field and post-hoc analysisof “pseudo-composite” images.

Logical processes can be used for detection of regions of profoundlyhypofluorescent/dark DNIRA signal.

Software and hardware unit can be used for the unprocessedhigh-resolution DNIRA images undergo registration, alignment andsegmentation. In cases where baseline (pre-ICG injection) images havedemonstrable signal prior to ICG injection, this signal is subtractedfrom all other images. Evaluated at a single timepoint, regions ofprofoundly hypofluorescent/black DNIRA signal are identified andquantified. Evaluated over several timepoints, images are registered andaligned, and the rates of change identified. DNIRA has the capability todetect black/hypofluorescent signal that can be subsequently analyzedand followed over time.

Example 2: Total Hypofluorescent DNIRA Areas as a Marker of Disease andDisease Burden (Diagnostic or Prognostic Biomarker)

FIG. 24 is a graphical representation of this example.

The “Total Hypofluroescent DNIRA Area(s)” may reflect total burden ofdisease, at a given point in time. The hypofluorescent DNIRA signal isreadily segmentable and quantifiable, providing the first readilytractable measure of disease in addition to GA (determined by FAF, notshown here).

Subject consent is used for all clinical research protocols are ResearchEthics Board (REB) approved and subjects willing and able to consent areserially enrolled.

Baseline in vivo imaging: baseline DNIRA images are acquired using acommercially available confocal scanning laser ophthalmoscope (cSLO)(e.g. Heidelberg Retinal Angiography, HRA-2, Heidelberg Engineering,Germany). Images are obtained prior to systemic injection using cSLO,and those with significant optical opacity are eliminated from furtherimaging (other than as clinically indicated). Amongst subjects withsuitable optical clarity, additional images are also obtained in thered-free, FAF (488/500 nm excitation/emission), IR reflectance channel(830 nm) and ICG fluorescence channels (795/810 nm excitation/emission),along with optical coherence tomography (OCT) and color fundusphotography. Within a suitably short time-frame, for example between 1and 28 days, subjects undergo ICG injection that may or may not includeangiography.

ICG injection: ICG dye (Cardiogreen, Sigma) is freshly prepared prior toexperimentation to a final stock concentration of 5.0 mg/ml in sterilewater. A fine gauge catheter is inserted intravenously, and ICG dye isinfused. If angiography is performed, images are acquired prior toinjection (baseline), during dye circulation, and at various intervalsthereafter typically out to 40 minutes or 2 hours.

DNIRA imaging: Within a suitable time-frame, for example 1 to 5 days(but potentially 24 hours to 120 days after ICG), subjects return forDNIRA images obtained with ICG excitation/emission filters or lasersystems in place (795/810 nm) but without further injection of dye.Multiple images are acquired of each eye including those centered on thefovea and the optic nerve head. Images are also taken in four quadrantsof the macula, thereby permitting visualization of a larger field andpost-hoc analysis of “pseudo-composite” images.

Repeated DNIRA Imaging: to determine the temporal changes between DNIRAimages over time, subjects return for repeated testing for example at 1,2, 3, 4, 6, 8 or 12 month intervals, or longer.

Logical processes can be used for a temporal analysis of regions ofexpanding hypofluorescent DNIRA signal. Logical processes can also beused for an inter-modality analysis of regions of expandinghypofluorescent DNIRA signal.

The left pair of images show a patient who has a relatively small TotalHypofluroescent DNIRA Area. The right pair of images show a patient whohas a relatively larger Total Hypofluorescent DNIRA Area. Comparing thetwo, a ratio or proportion can be established. In this case, the patienton the right has over 3 times greater identified as hypofluorescentusing DNIRA. Such total areas can also be measured over time to providea measure of progressive or dynamic change.

Example 3: Total Hypofluorescent DNIRA Areas as a Marker of Progressiveand Dynamic Change (Monitoring, Predictive or Prognostic Biomarker)

FIGS. 17A and 17B are graphical representations of this example.

The “Total Hypofluroescent DNIRA Area(s)” may reflect total burden ofdisease, at a given point in time. The hypofluorescent DNIRA signal isreadily segmentable and quantifiable, providing the first readilytractable measure of disease in addition to GA (determined by FAF, notshown here).

Subject consent is used for all clinical research protocols are ResearchEthics Board (REB) approved and subjects willing and able to consent areserially enrolled.

Baseline in vivo imaging: baseline DNIRA images are acquired using acommercially available confocal scanning laser ophthalmoscope (cSLO)(e.g. Heidelberg Retinal Angiography, HRA-2, Heidelberg Engineering,Germany). Images are obtained prior to systemic injection using cSLO,and those with significant optical opacity are eliminated from furtherimaging (other than as clinically indicated). Amongst subjects withsuitable optical clarity, additional images are also obtained in thered-free, FAF (488/500 nm excitation/emission), IR reflectance channel(830 nm) and ICG fluorescence channels (795/810 nm excitation/emission),along with optical coherence tomography (OCT) and color fundusphotography. Within a suitably short time-frame, for example between 1and 28 days, subjects undergo ICG injection that may or may not includeangiography.

ICG injection: ICG dye (Cardiogreen, Sigma) is freshly prepared prior toexperimentation to a final stock concentration of 5.0 mg/ml in sterilewater. A fine gauge catheter is inserted intravenously, and ICG dye isinfused. If angiography is performed, images are acquired prior toinjection (baseline), during dye circulation, and at various intervalsthereafter typically out to 40 minutes or 2 hours.

DNIRA imaging: Within a suitable time-frame, for example 1 to 5 days(but potentially 24 hours to 120 days after ICG), subjects return forDNIRA images obtained with ICG excitation/emission filters or lasersystems in place (795/810 nm) but without further injection of dye.Multiple images are acquired of each eye including those centered on thefovea and the optic nerve head. Images are also taken in four quadrantsof the macula, thereby permitting visualization of a larger field andpost-hoc analysis of “pseudo-composite” images.

Repeated DNIRA Imaging: to determine the temporal changes between DNIRAimages over time, subjects return for repeated testing for example at 1,2, 3, 4, 6, 8 or 12 month intervals, or longer.

Logical processes can be used for a temporal analysis of regions ofexpanding hypofluorescent DNIRA signal. Logical processes can also beused for an inter-modality analysis of regions of expandinghypofluorescent DNIRA signal.

In FIG. 17A, The “Total Hypofluorescent DNIRA Area” may can be monitoredover time, and compared repeatedly, against a specified starting time(baseline). Areas of profoundly hypofluorescent DNIRA can be segmented,and the features extracted and compared over time. Other embeddedmetadata of the images can also be used to extract other features. Theexample shows 3 timepoints over time where regions of hypofluorescenceare segmented out (green trace) and compared against each other todetect changes at a given timepoint.

In FIG. 17B, the upper panel shows the green tracings illustratingborders of regions of profound hypofluorescent DNIRA (from images inmiddle panel), that unlike observations of GA, (detected using FAF orOCT), can become smaller in size over time. These data suggest a muchmore dynamic and changing aspect of disease not previously recognized,and illustrate that regions outside GA or macular atrophy (MA) can alsobe readily quantified. DNIRA images, also demonstrate complex,non-segmentable patterns that represent unstructured, image-based datasuitable for cognitive or Artificial Intelligence (AI) based analytics.Other forms of metadata may also be used to describe the images. Thelower right image shows that dynamic changes can be compared over timeto provide a time-dependent measure. Such data can be used for diseaseprognosis, prediction, monitoring of patients over time, and helpidentify treatment strategies.

Example 4: Rates of Change of Total Hypofluorescent DNIRA Areas as aMarker of Prognosis or Response to an Intervention

FIG. 25 is a graphical illustration for this process.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

Logical processes were used for a temporal analysis of regions ofexpanding hypofluorescent AMDI signal. Logical processes were used foran inter-modality analysis of regions of expanding hypofluorescent AMDIsignal

The leftmost panel shows that this patient has early dry AMD, confirmedusing FAF an absence of Geographic Atrophy (GA) throughout the course ofstudy.

The middle and right panels show DNIRA, and green trace illustratesborders of regions of profound hypofluorescent DNIRA, that unlikeobservations of GA (detected using FAF or OCT), can become smaller insize over time. Compared with the patient shown in the previous example(Example 3, FIGS. 17A and 17B), the rate of change was more rapid, thuspermitting comparisons between groups of individual and potentiallypredicting more rapid transition to late AMD.

These features may be particularly useful in identifying responses to anintervention that takes place in this timeframe, as well as prognosis ofdisease progression.

Example 5: DNIRA Identifies Regions of Profound Hypofluorescent/BlackSignal That Extend Beyond the Regions of Hypofluorescent/Black FAF—TheCalculation of “Delta” as a Measure of Disease Burden

FIG. 14 is a graphical illustration of this concept.

DNIRA can identify regions of profound hypofluorescence/black signal inexcess of that observed using FAF, the present gold standard forevaluation of subjects with late dry AMD. FAF identifies regions ofRPE/photoreceptor loss known as GA (where the endogenous fluorophoressuch as lipofuscin are absent). When present, areas of GA always liewithin areas of absent DNIRA signal. As such, logical and softwarealgorithms have been developed to identify and quantify the extent ofabsent DNIRA signal in excess of absent FAF signal. This can represent ameasure of the burden of disease, rather than areas where tissue islost. FIGS. 15A and 15B are graphical illustrations for this example.

Subject consent: all clinical research protocols are Research EthicsBoard (REB) approved and subjects willing and able to consent areserially enrolled.

Baseline in vivo imaging: baseline DNIRA images are acquired using acommercially available confocal scanning laser ophthalmoscope (cSLO)(e.g. Heidelberg Retinal Angiography, HRA-2, Heidelberg Engineering,Germany). Images are obtained prior to systemic injection using cSLO,and those with significant optical opacity are eliminated from furtherimaging (other than as clinically indicated). Amongst subjects withsuitable optical clarity, additional images are also obtained in thered-free, FAF (488/500 nm excitation/emission), IR reflectance channel(830 nm) and ICG fluorescence channels (795/810 nm excitation/emission),along with optical coherence tomography (OCT) and color fundusphotography. Within a suitably short time-frame, for example between 1and 28 days, subjects undergo ICG injection that may or may not includeangiography.

ICG injection: ICG dye (Cardiogreen, Sigma) is freshly prepared prior toexperimentation to a final stock concentration of 5.0 mg/ml in sterilewater. A fine gauge catheter is inserted intravenously, and ICG dye isinfused. If angiography is performed, images are acquired prior toinjection (baseline), during dye circulation, and at various intervalsthereafter typically out to 40 minutes or 2 hours.

DNIRA imaging: Within a suitable time-frame, for example 1 to 5 days(but potentially 24 hours to 120 days after ICG), subjects return forDNIRA images obtained with ICG excitation/emission filters or lasersystems in place (795/810 nm) but without further injection of dye.Multiple images are acquired of each eye including those centered on thefovea and the optic nerve head. Images are also taken in four quadrantsof the macula, thereby permitting visualization of a larger field andpost-hoc analysis of “pseudo-composite” images.

Logical processes were used for elimination of background signal. Insome cases, where autofluorescent signal is obtained in the baselineDNIRA image (e.g., prior to dye injection), such baseline signal ismathematically eliminated prior to further image analysis.

Logical processes were used for calculation of “delta” the differencebetween hypofluorescent DNIRA and FAF images. In cases where GA ispresent, DNIRA identifies regions of profound hypofluorescence/blacksignal that are equivalent or larger than areas of profoundhypofluorescent/black signal obtained using FAF.

Software and hardware unit for the unprocessed high-resolution DNIRA andFAF images to undergo registration, alignment and segmentation: In caseswhere baseline (pre-ICG injection) images demonstrate demonstrablesignal prior to ICG injection, this signal is subtracted from all otherimages. Evaluated at a single timepoint, the registered DNIRA and FAFimages are compared and the regions of hypofluorescent DNIRA that extendbeyond the boundary of dark FAF or that exist in regions distinct fromthe dark FAF, are identified (both in a separate image and as part of atwo-layer overlay) and quantified (following segmentation). In thisexample, DNIRA identifies multiple regions outside the FAFhypofluorescence that contain regions of black DNIRA signal, increasingthe total number of black hypofluorescent signals, total area of blacksignal, and the perimeter or edge of the region of black.

Example 6: DNIRA Features to Identify Early Disease and DifferentPhenotypes of Early Disease (Diagnostic Biomarker)

FIGS. 26A and 26B are graphical representations of this example.

DNIRA may distinguish different phenotypes of early AMD. These mayrepresent differences in underlying biology (genetic or environmental),potentially amendable to different and targeted therapies. Although AMDhas a strong genetic component, it accounts for only approximately up to½ of disease susceptibility. Amongst patients with a genetic background,the statistical likelihood that they may or may not develop disease canbe calculated for the population, but not for the individual. Bycontrast, early changes in DNIRA may confirm a positive inheritance,e.g. very early disease, even prior to the formation of drusen.

With subject consent, all clinical procedures were performed asdescribed in previous examples. DNIRA imaging procedure was carried outon a patient having no personal history or known family history ofdisease at the time of presentation. However, in subsequent questioninga positive family history was obtained in the time between clinicalvisits

In FIG. 26A, the left panel shows a case patient with family history ofAMD but no personal diagnosis (absence of drusen was confirmed). DNIRAimage shows complex, albeit faint hypofluorescent pattern. The rightpanel shows a decade-matched control patient with no family or personalhistory of AMD, and no drusen found. The absence of AMD using currentclinical definitions (absence of drusen) was confirmed by clinicalexamination. This exemplifies the ability of DNIRA to detect subtlechanges in patients that may otherwise not present with any identifiablefeatures of disease.

In FIG. 26B, the upper panel shows two patients who have a diagnosis ofearly AMD based on clinical examination. On the left is a DNIRA image ofpatient with early AMD that shows marked dark grey, loose weave pattern.On the right is a DNIRA image that shows a subtle loose weave pattern.Corresponding FAF images confirm that these two patients have early dryAMD, however show relatively little difference with this modality. DNIRAand FAF images were obtained at the same timepoint.

Thus, DNIRA patterns observed in early AMD may assist in makingpersonalized diagnose of AMD earlier than other methods, allowing forunique and targeted therapies.

Example 7: DNIRA Features to Identify Disease Subtypes Among PatientsWith a Known Diagnosis of AMD and to Correlate with Disease Pathogenesis(Predictive Biomarker)

FIGS. 21A-21E are graphical representations of this example.

It is known that blinding diseases such as AMD are complex, resultingfrom both multigenic inheritance (over 30 genes and 200 SingleNucleotide Polymorphisms) that is subject to significant environmentaloverlay, making successful treatment difficult. Novel groupings ofdisease and disease features correlate with complex inherited andepigenetic influences.

DNIRA and related methods can identify previously unseen features andcomplex new phenotypes of disease. For example, several complex 2Dpatterns, labeled “tight weave”, “loose weave”, “grey smudge”, “oilstains”, “bullet holes”, etc. have been identified. As such, DNIRA canbe used to define new phenotypes and thereby subsequent novelgenotype-phenotype correlations.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

FIGS. 21A-21E depicts examples where DNIRA uniquely identifies novelimage-based phenotypes, exemplified by the currently identifiedphenotypes termed: (FIG. 21A. “loose weave” pattern, FIG. 21B. “leopardspots”, FIG. 21C. “grey smudge”, FIG. 21D. “fingerling potatoes”, andFIG. 21E. “very large bright spots.”

These types of complex phenotypes are previously unknown and have notbeen identified using any other imaging modality, depicting the uniqueability of DNIRA to capture them.

Example 8: DNIRA Features to Distinguish Diseases of Similar ClinicalAppearance to Assist with Diagnostic Accuracy (Diagnostic Biomarker)

FIGS. 27A and 27B are graphical representations of this example.

Complex DNIRA patterns observed in association with other disorders,with no personal history of AMD, may provide an early diagnosticbiomarker. For example early Vitelliform Disease, particular adult onsettype (AVD), is characterized by central foveal lesions often mistakenfor AMD. In some cases, the diagnosis is made following a non-responseto anti-VEGF agents (prescribed for late neovascular “wet” AMD).However, as a potential diagnostic biomarker, DNIRA identifies complex2D patterns of hypo/hyper-fluorescence not observed amongst patientswith AVD. Another non-AMD disease that can be confused with AMD isCentral Serous Retinopathy (CSR).

In the cases described here, with subject consent, all clinicalprocedures were performed as described in previous examples.

In FIG. 27A, the upper panels show images of the right eye of a patientwith visual complaints and a tentative diagnosis of AMD owing to thepresence of small, yellowish material in the central fovea (color imagenot shown). IR and FAF are unremarkable, while DNIRA shows a faint,variable appearance. The lower panels show images of the left eye ofthis patient, and confirm the diagnosis of AVD, with the appearance ofan apparent “fluid line” across the central lesion. The middle lowerpanel shows DNIRA image with hypofluorescence in the area of the centrallesion, with little background contrast (as noted in the other eye).This contrasts starkly with the DNIRA image obtained from a patient withearly AMD (lower left) where a weave pattern is evident.

In FIG. 27B, a non-AMD patient with CSR in the left eye, showed acomplex “loose weave” pattern in both eyes despite no family history norpersonal history of AMD and no drusen in either eye. Over the course ofstudy, drusen were detected in the non-CSR eye confirming a diagnosis ofAMD and suggesting that DNIRA may be used as an early diagnostic toolfor AMD (unable to evaluate in the afflicted eye due to CSR-induceddamage).

The upper panel of B shows the right eye is normal using Infra-Red andFAF, but DNIRA shows “loose weave” pattern. The middle panels show theleft eye that has both features of CSR and the weave. The lower panelshows color also confirm a normal eye exam, however OCT confirmeddevelopment of drusen in follow-up evaluation after this patentcompleted the study (approximately 2 years after 1st exam). The firstOCT image taken in 2015 demonstrates that absence of drusen in a patientwith no personal or family history of AMD, while the one below of thesame patient in 2017 demonstrates the development of classic drusen,confirming the diagnosis of AMD. This diagnosis were surmised in 2015 bythe presence of the “weave” pattern.

This exemplifies that DNIRA reveals some complex image-based patternsthat are absent in patients with ocular conditions other than AMD, orfrequently misdiagnosed as AMD, such as but not limited to AdultVitelliform disease, Bietti's crystalline dystrophy, pattern dystrophy,and others.

Thus, DNIRA can be used to identify patients with disorders thatclinically mimic AMD, may be used as a diagnostic biomarker to detectearly eye disease and enable early therapeutic, lifestyle or environmentintervention.

Example 9: DNIRA Features Identify Inherited Monogenic Disorders andDisease that May Not Yet Be Diagnosed (NYD)

FIGS. 28A and 28B are graphical representations of this example.

Complex DNIRA patterns may provide an early diagnostic biomarker forinherited monogenic disorders such as Stargardt disease, retinitispigmentosa, Leber congenital amaurosis (LCA), or Bardet-Biedl syndrome(BBS).

In this example, a patient with inherited, monogenic disorder (Stargardtdisease) illustrates the ability of DNIRA to detect disease, and detectregions at potential risk of disease progression from early to late inthese types of disease.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

In FIG. 28A (left panel), the left and middle images show IR and FAFimage of the left fundus of a patient with Stargardt disease, wherebright, hyperfluorescent lesions with the characteristic “pisciform”shape can be observed. The right image shows DNIRA, where regions ofprofound hypofluorescence that are both segmentable and surmised torepresent areas of poor dye uptake or accumulation, and/or potentiallyreduced choroidal flow can be observed. These are distinct fromNIR-autofluorescence (NIR-AF) a method used in the evaluation ofStargardt disease that utilizes NIR light but without the earlierdelivery of a NIR dye (so detecting autofluorescence from pigments suchas melanin).

The enlarged yellow box (rightmost image) shows DNIRA and identifiessmall hyperfluorescent bright dots in association with areas ofhypofluorescent DNIRA. These dots may represent actively phagocyticmacrophages that ultimately lead to macular atrophy (MA) in thisdisease. Throughout the image they are found particularly in associationwith regions of hypofluorescent DNIRA signal, potentially ascribingmacrophage activity to disease progression thus linking DNIRA with apotential treatment strategy (such as for example macrophage modulationor inhibition)

In FIG. 26B, a DNIRA image shows a presumed inherited eye disease thatmay share some aspects of genetic susceptibility with AMD, but has notyet been diagnosed (NYD) is shown.

Thus, DNIRA can be used to diagnose patients with disorders other thanAMD, may be used as a diagnostic biomarker to detect early eye diseaseand enable early therapeutic, lifestyle or environment intervention.

Example 10: DNIRA Feature to Monitor Progression of Disease or theEffect of a Treatment Over Time (Monitoring Biomarker)

FIG. 29 is a graphical representation of this example.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

Hypofluorescent features identified with DNIRA can be observed to expandover time. This is similar to observations made using FundusAutofluorescence (FAF) to quantify the rate of expansion of geographicatrophy (GA), more recently performed using en face OCT.

At present, in the absence of other useful markers to drive clinicaltrial design for the potential treatment of dry AMD, the rate of GAexpansion is an acceptable Phase III clinical trial endpoint. An exampleof this is shown in the top panel which depicts how FAF identifiesregions of hypofluorescent signal, which followed over time demonstratethe expansion of GA. The lower panel shows that DNIRA identifiessignificantly more hypofluorescent signal, which changes over time andidentifies more disease.

The faint purple lines have been used to trace the perimeter of theabnormal hypofluorescent FAF in the upper panel, and faint green linesto do the same in the lower DNIRA panel. These extracted image elementsare shown in the lower 2 panels below the fundus images.

Thus, DNIRA can be used to monitor disease progression, in many casesproviding more information than current methods such as FAF.

Example 11: DNIRA Feature to Monitor Disease Progression, ComparingWithin and Across Modalities and Other Imaging Biomarkers Over Time(Monitoring and Prognostic Biomarker)

FIG. 30 is a graphical representation of this example.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

Shown in the figure are cases observed to date where DNIRA was comparedacross timepoints, and to other imaging modalities to identify featuresthat can serve as monitoring and prognostic biomarkers.

FAF and DNIRA features were traced using purple and green linesrespectively, and have been used to trace the perimeter of the abnormalhypofluorescent FAF or DNIRA signal. These extracted image elements areshown alongside the fundus images.

This patient has Central Serous Retinopathy in the left eye, and atfirst visit, no drusen in either eye. However, a marked complex 2D“weave” pattern was observed in both eyes. Over the course of study,drusen were detected in the non-CSR eye (unable to evaluate in theafflicted eye). The leftmost panel shows extensive regions ofhypofluorescent DNIRA observed at baseline (DNIRA S1) in this patient.

The right two upper panel images show baseline and 8 month follow-up FAFimages.

The lower panels show registered and segmented DNIRA (green outlines)and FAF (purple outlines) images taken over time and show progression ofGA into regions of hypoDNIRA signal.

Comparing across timepoints DNIRA session 1 (S1) can be observed todetect features that are only apparent on FAF session 3 (S3), indicatingthat DNIRA can be used to monitor disease progression.

The expansion of GA into hypofluorescent regions using DNIRA is observedin all patients to date. By contrast, not all areas of hypofluorescentDNIRA become areas of GA (at least over the time course of this study)

Therefore, regions of GA extend only into areas of pre-existenthypofluorescent DNIRA. Thus, some regions of hypofluorescent DNIRAsignal can be used to predict regions of GA expansion.

Example 12: DNIRA Feature as a Biomarker to Identify Patients Likely toProgress to Later Disease (Prognostic Biomarker)

FIG. 31 is a graphical representation of this example.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

Shown in the figure is a case where DNIRA was compared acrosstimepoints, and to other imaging modalities to identify features thatcan serve as prognostic biomarkers where specific features and complexpatterns observed using DNIRA may predict disease progression.

In some cases, these features may be not previously described, while inothers, they may correlate with previously known risk-factors fordisease progression such as the presence of large RPE detachments.

Exemplified is a patient with early dry AMD The left panel shows earlyAMD using FAF with no GA, indicated by absence of regions of abnormalprofound hypofluorescence. The right panel shows the same region, imagedat the same timepoint with DNIRA. Here, DNIRA identifies multiplefeatures of disease including both quantifiable regions of profoundhypofluorescence and complex 3D patterns.

The middle panel depicts a second case, where heat map analysisperformed at three timepoints (at a 4-month intervals) shows thatcompared against the DNIRA image (green) at Session 1 (S1, left), thecorresponding FAF image (red) is smaller, but progressively “fills in”the region identified as vulnerable using DNIRA. By 8 months, theregions of GA identified using FAF expand beyond the regions originallypredicted by DNIRA (noting that the DNIRA map also enlarged such thatevaluated at the same timepoint, the GA signal remains within, iesmaller than, the DNIRA signal).

The lower panel shows (8 months later) simultaneous FAF and DNIRAimaging confirms that the DNIRA region also expanded over time and thatthe region of GA identified using FAF falls within its borders (theapparent extension of the FAF signal inferior to the merged signalreflects that ill-defined or “fuzzy” border of area of GA obtained usingFAF)

Thus, DNIRA signal can be used to predict regions of GA expansion.

Example 13: DNIRA Feature as a Biomarker to Quantify Aspects of DiseaseKnown to Predict Progression (Prognostic Biomarker)

FIG. 32 is a graphical representation of this example.

DNIRA can readily identify, quantify and follow over time thespatiotemporal changes in disease features predicted to correlate withdisease but currently not quantifiable and hence not demonstrated. Forexample, drusen were identified as a key prognostic factor predictingthe 10 year risk of patients progressing to blinding late AMD (both wetand dry). The clinical utility of large soft and soft confluent drusenprecludes their application in prevention studies owing to the largesample size necessary to adequately power a study. Large and mediumdrusen confer disease risk and risk of progression, but their dynamicchanges over time are only recently described. Not readily quantifiable,it is currently not possible to provide a measure of their change.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

The figure depicts large soft and confluent soft drusen identified asyellowish circular/oval deposits observed on color fundus photography(CFP) in the upper panel. The lower panel correlates the same region ofthe image using DNIRA, depicting clearly demarcated areas of black wherethe drusen are localized.

Thus, DNIRA can readily identify and quantify known high-risk prognosticfeatures of disease such as large, soft and confluent drusen.

With that, we describe a Dynamic Drusen Index (DDI) that can calculatetotal drusen burden at single or multiple timepoints, or can calculatethe change in particular drusen subtypes. Relying on the currentclassification of drusen, and the utility of novel DNIRA images toidentify drusen, we suggest Total DDI (for all drusen types), LargeConfluent Dynamic Drusen Index (LC-DDI), Large Dynamic Drusen Index(L-DDI), Medium Dynamic Drusen Index (M-DDI), and Small Dynamic DrusenIndex (S-DDI). DDI can instruct which patients to preferentially enlistin particular clinical trials, thereby serving a Predictive Biomarker

Example 14: Using DNIRA to Correlate with Disease Pathogenesis(Predictive Biomarker)

FIG. 33 is a graphical representation of this example.

Drusen were identified as a key prognostic factor predicting the 10 yearrisk of patients progressing to blinding late AMD (both wet and dry).The clinical utility of large soft and soft confluent drusen precludestheir application in prevention studies owing to the large sample sizenecessary to adequately power a study.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

To correlate DNIRA in patients to a valid model that depicts macrophagesin the eye, retinal atrophy was induced in rabbits using systemicinjection of oxidative agent, and imaged using the DNIRA method. DNIRAimages were acquired using a commercially available confocal scanninglaser ophthalmoscope (cSLO) (e.g. Heidelberg Retinal Angiography, HRA-2,Heidelberg Engineering, Germany). Images are obtained prior to systemicinjection using cSLO, in the red-free, FAF (488/500 nmexcitation/emission), IR reflectance channel (830 nm) and ICGfluorescence channels (795/810 nm excitation/emission). ICG dye(Cardiogreen, Sigma) is freshly prepared prior to experimentation to afinal stock concentration of 5.0 mg/ml in sterile water. A fine gaugecatheter is inserted intravenously into the marginal ear vein, and ICGdye is infused. Within 5-14 days after toxin injection, and 2-3 daysafter dye injection, DNIRA images obtained with ICG excitation/emissionfilters or laser systems in place (795/810 nm) but without furtherinjection of dye. Multiple images are acquired of each eye, including atthe center and in the periphery.

The image on the left shows a DNIRA image of a patient whichdemonstrates a ring of hyperfluorescent dots that correspond withpresumptive macrophages positioned just beyond a region ofhypofluroescence.

The middle image shows DNIRA images of a rabbit eye following acuteoxidative stress demonstrate similar arrays of brightly hyperfluorescentdots around hypofluorescent regions.

After imaging, rabbit eyes are removed for immunofluorescent tissueanalysis using markers for macrophages, such as Iba-1.

The right panel shows immunofluroscent labelled rabbit retinal tissue,with the presence of Iba-1 positive macrophages (red) surrounding theregions of outer retinal damage and RPE loss (blue nuclear stain).

Further, the lower panel confirms the distribution of Iba1+ macrophagessurrounding small, newly formed areas of outer retinal damage (depictedby rings and convolutions of the outer nuclear layer (blue nucleistaining) and RPE loss (depicted by the sparse staining of RPE markerRPE65 in green) analogous to geographic atrophy.

Thus, DNIRA can identify presumptive macrophages in the eyes of patientswith diseases such as AMD or ocular tumors, identifying individuals whoare more likely than similar individuals without the biomarker toexperience a favorable or unfavorable effect from exposure to a medicalproduct such as macrophage modulating therapy or an environmental agent.

Example 15: Analysis of Tumor Associated Macrophages (TAMs) Using DNIRA(TAMI) as Predictor of Disease Severity in Ocular Melanoma

FIGS. 18 and 19A-J are graphical representations of this example.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

Multimodal and cross-modal analysis for identification of regions ofinterest (ROIs) containing hyperfluorescent TAMI signal. Logical processfor temporal analysis of discrete, dot or spot-like, hyperfluorescentTAMI signal. Logical process for temporal analysis of aggregates oraccumulations of dot or spot-like TAMI signal. Logical process forinter-modality analysis of hyperfluorescent dots or spots.

FIG. 18 shows a comparison of DNIRA vs colour photos & clinical gradingof uveal melanoma examples. The left two images show uveal nevi, middleright image shows an indeterminate lesion, and the right image shows amelanoma image. In the upper panel DNIRA images show the absence orpresence of bright hyperfluorescent dots in the tumour area, or area ofassociated fluid. These images were used to perform analyses describedin FIGS. 19A-G.

FIGS. 19A-G show TAMI composite images, for correlation of the TAMIimaging with currently available clinical imaging modalities. Assemblyof TAMi composites allow for correlation of the novel imaging methodwith current clinical modalities. Fundus autofluorescence (FAF, FIG.19B) and TAMi composites (FIG. 19C) were assembled as an overlay oncolour photographs (FIG. 19A). Multi-modal analysis allowedidentification of regions of interest including the tumour region (i),areas of current or previous fluid (ii), and peripheral (extra-lesional)regions (iii). FIG. 19D depicts multimodal analysis allowingidentification of a tumor region. FIG. 19E depicts multimodal analysisallowing identification of a fluid region. FIG. 19F depicts identifiedregions of interest, including a tumor region (i), an area of current orprevious fluid (ii), and an area of peripheral (extra-lesional)features. FIG. 19G depicts identification of regions of interest fromcomposite images. Region area in mm² was determined using ImageJmeasurements and used in hyperfluorescent dot density calculations (FIG.19H). These were compared across patients with melanomas, indeterminate74 lesions, or benign nevi, as determined by clinician ground truth.

Comparison to subject risk features including orange pigment presence,lesion area on color fundus photos, lesion thickness, lesion elevation,proximity to optic nerve head, presence of subretinal fluid, presence ofhot spots, and degree of pigmentation. The risk features were analyzedin subjects with benign nevi, ocular melanoma, and control subjects, andcompared to hyperfluorescent dot counts observed with TAMI/DNIRAimaging. Mean regional dot density is shown as number of dots/mm²±SEM.

(FIG. 19I) One-way ANOVA and post hoc multiple caparisons showsignificantly higher dot densities in melanoma arms when consideringlesion and fluid regions, but not in peripheral regions. (FIG. 19J)Multiple comparisons of dot density in each region within risk groupsfound melanomas had significantly higher regional dot density in lesionand fluid regions when compared to peripheral areas, and this was notobserved in other risk groups.

Presence of hyperfluorescent dots was correlative to presence of ocularmelanoma, but not to benign nevi, or control subjects, suggesting thathyperfluorescent dot presence can predict severity of ocular tumors.

Example 16: DNIRA to Detect Multiple Types of Hyperfluorescent DotSignals (Predictive Biomarkers)

FIGS. 34A-34D are graphical representations of this example.

DNIRA has the potential to identify patients with increased presence ofintraocular macrophages, serving as a predictive biomarker, and providea unique link between imaging biomarkers and disease pathobiology. Sinceconfirmatory ocular biopsy from human subjects is often not possible,analysis of these features were previously correlated to preclinicalstudies.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

Repeated analysis, exemplified here from just two patients, shows thesize distribution of hyperfluorescent dots, demonstrated to bemacrophages in correlative pre-clinical studies.

In FIG. 34A, the left panel shows small hyperfluorescent dots that canbe seen in some patients using DNIRA. These have high degrees ofcircularity and are of approximately the same size. The middle panelshows another type of bright dots that have dark centers. The rightpanel shows very large bright dots that are observed in some forms ofdisease.

In FIG. 34B, Kruskal-Wallis analysis shows that size and uniformity ofthe dots is consistent with their cellular identity. The distributionsof dot size are fairly similar—both qualitatively (histograms) andquantitatively (Kruskal-Wallis p-value 0.7577 indicates no significantdifference). Mean size in pt 005 is 399 μm², with a mean diameter of22.5 μM; mean size in pt 010 is 483 μm², with a mean diameter of 24.8μM. Macrophages in the living organism are predicted to range from 15 to40 μm in diameter, depending on their activation status and otherinfluences, indicating these dots are consistent with macrophage size.

In FIG. 34C, distributions of dot circularity are similar acrosspatients—both qualitatively (histograms) and quantitatively(Kruskal-Wallis p-value 0.1055 indicates no significant difference).Mean circularity in pt 005 is 0.926; mean circularity in pt 010 is0.888. These values are consistent with measurements from pre-clinicalstudies in the rabbit eye.

Finally, in FIG. 34D, in pre-clinical investigation using a macrophagemodulator [such as, but not limited to bindarit and its derivatives,methotrexate, and others], we demonstrate that macrophage numbers, sizeand location are modified in treated animals compared to untreatedcontrols, suggesting the utility of DNIRA to not only identify patientsfor clinical trial but to monitor drug response. Thus, DNIRA is a usefulpharmacodynamic marker demonstrating significantly reduced signalfollowing macrophage modulatory therapy in pre-clinical testing in therabbit eye. It is anticipated that a similar reduction inhyperfluorescent signal may be observed in patients receivingmacrophage- and immune-modulating drugs.

Example 17: DNIRA to Detect Static and Dynamic Hyperfluorescent DotSignals (Predictive Biomarkers)

FIG. 35 is a graphical representation of this example.

DNIRA images show hyperfluorescent (bright) dots that are ephemeral ortransient in nature. Whether these dots have moved, disappears andreappeared, or whether new dots have arrived is not known.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

The upper panel on the left shows a pair of images comparing DNIRAagainst FAF. DNIRA shows additional areas of hyperfluorescent signal,plus bright, hyperfluorescent dots found in association with theseareas. In the right pair of images, the same set obtained at least threemonths later show subtle changes.

The middle panel shows two DNIRA fundus image of patient with Stargardtdisease, taken at least 3 months apart. Magnification of these imagesindicates bright dots that are present at both time points (bluearrows). At other sites bright dots have appeared in the second visitthat were not noted in the first (yellow arrows). The blue outline arrowin the right image indicates a site where a dot that was previouslythere moved, disappeared or become potentially less phagocytic and thusno longer takes up dye.

Example 18: DNIRA Detects Regions of Tissue Damage and MacrophageActivity in Central Serous Retinopathy (Predictive, PrognosticBiomarkers)

FIG. 36 is a graphical representation of this example.

Central Serous Retinopathy (CSR) is a disease characterized by fluidmovement from the choroid to the subretinal and intraretinal space. Withtime, persistent fluid is associated with irreversible chorioretinalatrophy (MA).

With subject consent, all clinical procedures were performed asdescribed in previous examples.

Images of a patient with CSR in the upper panel shows IR, FAF and DNIRAimages from left to right respectively. The FAF shows hyperfluorescencein area of previous (and potentially present) fluid. DNIRA similarlyidentifies region associated with fluid and also regions of decreasedfunctional ability to uptake dye, appearing as black (hypofluorescent).These regions suggest vulnerability to future tissue loss (not yetevident as areas of GA using the standard method of FAF).

In the lower panel the left, image taken in the ICG channel prior tosystemic dye injection has no signal. Middle and right images show thathyperfluorescent dots are found spatiotemporally within areas ofCSR-associated fluid and tissue damage.

These data may suggest that macrophages contribute to tissue damage, andboth macrophages and tissue damage can be detected using DNIRA, allowingfor earlier detection and prognosis.

Example 19: DNIRA Detects Two Populations of Bright HyperfluorescentDots in Diffuse-Tricking AMD (Diagnostic, Prognostic Biomarkers)

FIG. 37 is a graphical representation of this example.

Diffuse Trickling form of late dry AMD is typically defined by a uniqueFAF pattern. Here we identify novel biomarkers of diffuse tricklingsubtype of AMD.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

In the upper panel, left and middle images show IR and FAF respectively,identifying extensive GA. The right image shows DNIRA, with regions ofhypofluorescence that are of approximately the same area FAF, but alsoenlarged hyperfluorescent dots at the border.

The lower panel shows NIR imaging prior to the systemic delivery of ICGdye on the left, which shows no detectable signal. The middle imageshows the same DNIRA image as above, and enlargements of specific areas.On the right, DNIRA demonstrates two populations of hyperfluorescent(bright) dots at the border and in the surrounding so-called “junctionalzone”. Some are of typical size as presumptive macrophages (e.g., asseen in association with tumours), while others are much larger,potentially pointing to accumulation of dye within RPE cells, or othercells.

Example 20: DNIRA to Detect Reticular Pseudodrusen

FIG. 38 is a graphical illustration for this example.

Reticular pseudodrusen (RPD) is associated with particularly aggressivesubtypes of dry AMD, and a higher rate of disease progression.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

DNIRA image and subsequent logical processes of a periphery of thefundus can be used to show the presence of discrete, dot or spot-like,hyperfluorescent signal. This signal can be used for temporal analysisof aggregates or accumulations of dot or spot-like DNIRA signal(indicated by arrows); and for inter-modality analysis ofhyperfluorescent dots or spots.

This signal can subsequently be used to associate with reticularpseudodrusen (RPD) load, or disease burden, allowing for a potentiallybetter predictive biomarker of disease progression.

Example 21: DNIRA to Detect Ocular Inflammatory Disease

FIG. 39 is a graphical illustration for this example.

DNIRA can be applied to the analysis of ocular inflammatory disease. Inthis example of presumed Acute Multifocal Posterior PlacoidEpitheliopathy (AMPPE), DNIRA demonstrates abnormal RPE/outer retinalayer uptake of dye.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

In the left panel IR imaging shows variable reflectivity, includinghyper- and hypo-reflective regions, and puntacte signals. In the middlepanel, FAF similarly demonstrates hyper- and hypofluorescent regions,with regions of hyperfluorescence presumed to correlate with areas ofpotential disease activity. In the right panel DNIRA imaging showsmultiple regions of profound hypofluorescence which likely correlatewith areas of diseased tissue unable to accumulate dye, or potentiallyblocking its signal. Further, as shown within the box, a small area ofhypofluorescence is forming (oval area), present only in the DNIRA imagewith little change in other channels. Within this region are two largehyperfluorescent dots. Based on pre-clinical studies, these dotscorrespond with phagocytic macrophages, and are of consistent size andshape as dots seen in other conditions such as ocular melanoma.

Therefore DNIRA outlines larger areas of damage than either FAF or IRimaging in ocular inflammatory disease, and can be used as a diagnosticor prognostic biomarker.

Example 22: DNIRA as a Marker of High Risk AMD Features (Predictive,Prognostic Biomarker)

FIG. 40 is a graphical illustration for this example.

DNIRA can be used, for the first time, to quantify and bettercharacterize image-based features of diseases such as AMD, some of whichare already known to confer high risk of progression from early to latedisease (either atrophic or angiographic). Some of these features maycorrelate with histological features such as basal laminar deposits,basal linear deposits, or the deposition of extracellular, immune andill-defined materials deposited sub-retinal space or in the sub-RPEspace. Further, some deposits such as hyper-reflective material (in thesub-retinal space it is known as Subretinal Hyper-reflective Material,SRHM) are associated with high risk. However, much other material isobserved using OCT is not hyper-reflective and therefore not seen enface, and is poorly defined. As such it is not possible to quantify orcompare changes in these features over time.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

In the upper row, the left image shows an infrared image where variablesignal can be observed, with hyper-reflective material believed tocorrespond with the accumulation of pigments such as melanin that areknown to fluoresce in the IR spectrum. The upper right image shows thatFAF confirms that this patient has early AMD, with no obvious GA, butalso with hyper-reflective material present.

The middle row shows a sample DNIRA image prior to dye injection with nosignal visible. The right image shows DNIRA obtained at the same sessionas the upper panel FAF and IR images, and shows regions of profoundhypofluorescence. Further, small punctate hyper- and hypo-fluorescenceare also observed, the former of which may correspond with dye-labeledmacrophages or other phagocytic cells able to uptake the systemicallydelivered dye.

The lower row shows an OCT image where shallow drusen, sub-retinalmaterial and a shallow sub-RPE cleft associated with profoundlyhypofluorescent DNIRA signal is observed.

Thus DNIRA has the ability to highlight features of early AMD that maycorrespond to histological components associated with early disease, andthose associated with a high risk of disease progression, serving as apredictive or prognostic biomarker.

Example 23: Using DNIRA to Detect and Quantify Known Risk Factors forDisease Progression

FIG. 65 is a graphical representation of this example.

Drusen were identified as a key prognostic factor predicting the 10 yearrisk of patients progressing to blinding late AMD (both wet and dry).Despite their potential clinical utility, particularly for clinicaltrial design, soft drusen cannot be quantified or adequately describedusing current modalities as so their application is highly limited. Assuch, it is widely considered that large numbers of patients would benecessary to adequately power a study.

With subject consent, all clinical procedures were performed asdescribed in previous examples.

Upper panel: left is a colour fundus photo showing soft drusen anddrusenoid RPE detachments. They are obviously ill-defined. The middleimage shows the area of soft drusen and RPEDs, with their outline inblack. IN the right image, these signals can be segmented, or derivedthrough the use of a general cross-modal training algorithm, with fusionof multiple modalities that includes confirmatory OCT analysis, prior tothe generation of the OCT-driven segmentation map.

Lower panel: correspond OCT confirms that the yellow drusen observedclinical have a classic conformation. By contrast, the DNIRA imageconfirms that these drusend and RPEDs have a functional effect on theability of systemically delivered dye to generate a signal, which isdetected by the detector and processed by the classifier. The output isa colour map as shown.

Example 24: Detector—Soft Drusen

In one example, detector 1200 may be configured and trained toautomatically detect regions of soft fuzzy drusen. FIG. 66 showsmulti-modal imaging of soft drusen. Detection of large areas of softdrusen is of clinical interest as the soft drusen may merge to form RPEdetachments.

Detector 1200 may be configured to process input data DNIRA image data,and optionally including multi-modal image, to detect regions of softdrusen. FIG. 67 shows a region of soft drusen as detected by anembodiment of detector 1200. The use of DNIRA image data allows theirregular borders of soft drusen to be identified. Significantly, thisembodiment of detector 1200 may be used to provide automatic detectionof functional behavior of retinal tissue, and automatic quantificationof functional loss that accompanies soft drusen.

FIGS. 68-69 show an analysis of soft drusen by the quantification ofsoft fuzzy drusen and RPEDs, illustrating the utility of DNIRA for softdrusent imaging. These are examples of quantifying the biological effectof soft fuzzy drusen and large RPEDs on the transfer of dye from thechoroid to the RPE/photoreceptor layer, that in turn, is the firstfunctional imaging method. Large soft fuzzy drusen are the single bestknown predictor of progression from early to late AMD (both wet and dry,so CNVM and GA), thus is may be advantageous to quantify theirbiological significance.

Example 25: Detector—Macrophages

In one example, detector 1200 may be configured and trained toautomatically detect TAMs. FIG. 70 shows multi-modal imaging of softdrusen. Detection of TAMs is of clinical interest as the presence of TAMcorrelates with the severity of ocular tumors.

Detector 1200 may be configured to process input data DNIRA image data,and optionally including multi-modal image, to detect regions containinghyperfluorescent TAMI signal. FIG. 71 shows a TAM as detected by anembodiment of detector 1200.

In another example, detector 1200 may be configured and trained toautomatically detect macrophages associated with AMD. FIG. 72 shows aDNIRA image in which macrophages appear as bright dots. Detection of themacrophages is of clinical interest as macrophage activity may lead toGA and other forms of macular atrophy, and to CNV and other forms ofneovascularization.

Detector 1200 may be configured to process input data including DNIRAimage data, and optionally including multi-modal image data, to detectthe macrophages in AMD. FIG. 73 shows macrophages as detected by anembodiment of detector 1200. Detector 1200 may, for example, implementFast Fourier Transform (FFT) band pass filtering to detect highfrequencies associated with the clustering of macrophages. Detector 1200may be trained to find and utilize other features.

Embodiments of detector 1200 configured to detect macrophages may beused to assist in the development of a drug that targets macrophages,e.g., by measuring the efficacy of the drug.

Example 26: Detector—Grey Smudges

In one example, detector 1200 may be configured and trained toautomatically detect regions of grey smudge. Detection of grey smudge isof clinical interest as they may indicate tissue dysfunction. Of note,tissue dysfunctional may be distinguished from tissue death, whichresults in a black signal. In one specific embodiment, detector 1200 isconfigured to perform a K-means clustering algorithm. In this case,detector 1200 converts a DNIRA image to a single long vector. Thisvector is divided into K intensity levels based on the nearest neighboralgorithm. A segmentation map is then reconstructed by reassigning thepixel to their cluster means. Regions of grey smudge may then identifiedby comparing the intensity levels of segments to predefined thresholds,or thresholds learned through training. FIG. 74 depicts the applicationof K-means (K=5) segmentation to a set of four DNIRA images, withregions of grey smudge shown in each image.

Complex 2D Patterns

Complex 2D patterns of grey-scale hypofluorescent signal may be observedin DNIRA images, including many patterns that are not observable usingconventional imaging modalities such as FAF. Without wishing to belimited by theory, and by way of non-limiting example, these complex 2Dpatterns may represent different subtypes of disease, different stagesof disease, different diseases, different likelihoods of progressingdifferent rates of progression, different prognoses, different responsesto treatment, different underlying biology, different concurrentdiseases, different concurrent medical therapies, and/or differentlifestyle choices (e.g., smoking).

Accordingly, in various embodiments, a detector 1200 may be configure todetect a given one of the various categories of 2D patterns. Further, invarious embodiments, a classifier 1000 may be configured to classify anobserved pattern into the various categories of 2D patterns.

FIG. 75 shows an example 2D pattern, as observed in an FAF image, aDNIRA image, and a DNIRA image with regions classified by an embodimentof a classifier 1000. As shown, the FAF image lacks clearly demarcatedregions of profound hypofluorescence/black signal. However, in theprocessed DNIRA image, distinct regions are observable.

In some embodiments, a classifier 1000 provides for identification andquantification of regions of intermediate or grey-scale DNIRA signal. Insome embodiments, a classifier 1000 reduces the complexity of an imagethrough hierarchical feature extraction and dimensionality reduction.

EQUIVALENTS

While the disclosure has been described in connection with exemplaryembodiments thereof, it will be understood that it is capable of furthermodifications and this application is intended to cover any variations,uses, or adaptations of the invention following, in general, theprinciples of the invention and including such departures from thepresent disclosure as come within known or customary practice within theart to which the invention pertains and as may be applied to theessential features hereinbefore set forth and as follows in the scope ofthe appended claims.

Those skilled in the art will recognize, or be able to ascertain, usingno more than routine experimentation, numerous equivalents to thespecific embodiments described specifically herein. Such equivalents areintended to be encompassed in the scope of the following claims.

INCORPORATION BY REFERENCE

All patents and publications referenced herein are hereby incorporatedby reference in their entireties.

The publications discussed herein are provided solely for theirdisclosure prior to the filing date of the present application. Nothingherein is to be construed as an admission that the present is notentitled to antedate such publication by virtue of prior invention.

As used herein, all headings are simply for organization and are notintended to limit the disclosure in any manner. The content of anyindividual section may be equally applicable to all sections.

1. A computer system comprising: a processor; a memory in communicationwith the processor, the memory storing instructions that, when executedby the processor cause the processor to: at a training phase, receivetraining data corresponding to a plurality of ocular images; performfeature extraction and feature selection to generate features based onthe training data to build a pattern recognition model; at aclassification phase, receive a plurality of ocular images correspondingto a plurality of imaging modalities; classify features of the pluralityof ocular images using the pattern recognition model.
 2. The computersystem of claim 1, wherein the pattern recognition model is at least oneof a convolutional neural network, machine learning, decision trees,logistic regression, principal components analysis, naive Bayes model,support vector machine model, and nearest neighbor model.
 3. Thecomputer system of claim 1 or claim 2, wherein the feature extractiongenerates a masked image of defined shapes.
 4. The computer system ofclaim 3, wherein the feature selection is based on at least one offocality of the defined shapes, a number of focal points per unit areaof the defined shapes, and a square root of an area of the definedshapes.
 5. The computer system of any one of claims 1 to 4, wherein thefeatures are defined by areas of hypofluorescence.
 6. The computersystem of any one of claims 1 to 5, wherein the features are defined byareas of hyperfluorescence.
 7. The computer system of any one of claims1 to 6, wherein the plurality of ocular images of the training datacorrespond to a plurality of imaging modalities.
 8. The computer systemof claim 7, wherein the training phase further comprises building apattern recognition model for each of the plurality of imagingmodalities.
 9. The computer system of any one of claims 1 to 8, whereinthe plurality of ocular images comprises a cross-section image.
 10. Thecomputer system of any one of claims 1 to 9, wherein the plurality ofocular images comprises an en face image.
 11. The computer system of anyone of claims 1 to 10, wherein the training phase further comprisesregistering the plurality of ocular images to a common coordinatesystem.
 12. The computer system of any one of claims 1 to 11, whereinthe training phase further comprises cross-modal fusion of the pluralityof ocular images to a common coordinate system.
 13. The computer systemof any one of claims 1 to 12, wherein the plurality of imagingmodalities comprises at least one of delayed near-infrared analysis(DNIRA), infra-red reflectance (IR), confocal scanning laserophthalmoscopy (cSLO), fundus autofluorescence (FAF), color fundusphotography (CFP), optical coherence tomography (OCT), OCT-angiography,fluorescence lifetime imaging (FLI).
 14. The computer system of any oneof claims 1 to 13, wherein the memory stores further instructions that,when executed by the processor cause the processor to: generate across-section segmentation map corresponding to an en face region of aneye, each segment of the cross-section segmentation map corresponding toa cross-section image at that region of the eye; classifying eachsegment of the cross-section segmentation map as a phenotype of one ofnormal, drusen, retinal pigment epithelium detachments (RPEDs),pseudodrusen geographic atrophy, macular atrophy, or neovascularizationbased at least in part on classification of the cross-section imagecorresponding to that segment using the pattern recognition model. 15.The computer system of any one of claims 1 to 14, wherein the pluralityof ocular images comprises multiple cross-section images correspondingto multiple time points and the memory stores further instructions that,when executed by the processor cause the processor to: generate, foreach of the multiple time points, a cross-section segmentation mapcorresponding to an en face region of an eye, each segment of thecross-section segmentation map corresponding to a cross-section image atthat region of the eye; classify each segment of each cross-sectionsegmentation map as a phenotype of tissue state of one of normal,drusen, retinal pigment epithelium detachments (RPEDs), pseudodrusengeographic atrophy, macular atrophy, or neovascularization, based atleast in part on classification of the cross-section image correspondingto that segment using the pattern recognition model; and generate a timeseries data model based on the cross-section segmentation map at each ofthe multiple time points.
 16. The computer system of claim 15, whereinthe time series data model is based at least in part on identifiedchanges in the cross-section segmentation maps over time.
 17. Thecomputer system of claim 15 or 16, wherein the time series data model isused to generate a visual representation of disease progression.
 18. Thecomputer system of any one of claims 15 to 17, wherein the time seriesdata model is based at least in part on elapsed time between themultiple time points.
 19. The computer system of any one of claims 1 to18, wherein the features selected comprise phenotypes of a userassociated with the plurality of ocular images.
 20. The computer systemof any one of claims 1 to 19, wherein the memory stores furtherinstructions that, when executed by the processor cause the processorto: correlate the features with stage or grade variants of binding eyedisease including Age Related Macular Degeneration (AMD), monogenic eyedisease, inherited eye disease and inflammatory eye disease.
 21. Thecomputer system of any one of claims 1 to 20, wherein the memory storesfurther instructions that, when executed by the processor cause theprocessor to: correlate the features with stage or grade variants ofcentral nervous system (brain) disease, including at least one ofdementia and Alzheimer's disease.
 22. A computer-implemented method fordetecting a phenotype, comprising: receiving a plurality of ocularimages corresponding to a plurality of imaging modalities; registeringthe plurality of ocular images to a common coordinate system;classifying features of each of the plurality of ocular images using apattern recognition model; identifying features of the image as one ormore phenotypes.
 23. The computer-implemented method of claim 22,wherein the pattern recognition model is a convolutional neural networkbuilt based on training data corresponding to a plurality of ocularimages and feature extraction and feature selection is performed togenerate features from the training data.
 24. The computer-implementedmethod of claim 22 or 23, wherein the feature extraction generates agreyscale image of defined shapes.
 25. The computer-implemented methodof claim 24, wherein the defined shapes include at least one of leopardspots, loose weave, grey smudge, and fingerling potatoes.
 26. Thecomputer-implemented method of claim 24 or 25, further comprisingcorrelating one or more of the defined shapes with the presence ofphagocytic immune cells.
 27. The computer-implemented method of any oneof claims 22 to 26, further comprising generating one or moredescriptors of characteristics of the identified phenotype, including atleast one of location, size, quantity and colour.
 28. Acomputer-implemented method for predicting tissue loss, comprising:receiving a plurality of ocular images corresponding to a plurality ofimaging modalities; registering the plurality of ocular images to acommon coordinate system; classifying features of each of the pluralityof ocular images using a pattern recognition model; predicting tissueloss based at least in part on the features.
 29. Thecomputer-implemented method of claim 28, wherein the plurality ofimaging modalities comprises at least one of delayed near-infraredanalysis (DNIRA), infra-red reflectance (IR), confocal scanning laserophthalmoscopy (cSLO), fundus autofluorescence (FAF), color fundusphotography (CFP), optical coherence tomography (OCT), OCT-angiographyand fluorescence lifetime imaging (FLI).
 30. The computer-implementedmethod of claim 28 or 29, wherein the plurality of imaging modalitiesincludes cross-section images and en face images.
 31. Thecomputer-implemented method of any one of claims 28 to 30, wherein thepattern recognition model is a convolutional neural network built basedon training data corresponding to a plurality of ocular images andfeature extraction and feature selection performed to generate featuresfrom the training data.
 32. The computer-implemented method of claim 31,wherein the feature extraction generates a masked image of definedshapes.
 33. The computer-implemented method of claim 31 or 32, whereinthe feature selection comprises one or more of an area of geographicalatrophy, a square root of area of geographical atrophy, focality ofgeographical atrophy, focality index of geographical atrophy, and rateof geographical atrophy expansion.
 34. The computer-implemented methodof any one of claims 28 to 33, wherein the predicting tissue loss isbased on time series forecasting to predict tissue loss based on a timeseries data model.
 35. The computer-implemented method of claim 34,wherein the time series data model is generated based on multiplecross-section segmentation maps generated for each of multiple timepoints and corresponding cross-section images, each of the cross-sectionsegmentation maps corresponding to an en face region of an eye, and eachsegment of the cross-section segmentation map corresponding to across-section image at that region of the eye classified as a phenotypeof one of pseudodrusen, normal, drusen, or geographical atrophy, basedat least in part on classification of the cross-section imagecorresponding to that segment using a convolutional neural network. 36.The computer-implemented method of any one of claims 28 to 35, whereinthe features selected comprise phenotypes of a user associated with theplurality of ocular images.
 37. The computer-implemented method of claim36, further comprising identifying the phenotypes as risk factors bycorrelating the phenotypes with a rate of tissue loss over time.
 38. Thecomputer-implemented method of any one of claims 28 to 37, wherein thepredicting tissue loss is based at least in part on non-image basedbiomarker data.
 39. The computer-implemented method of claim 38, whereinthe non-image based biomarker data comprises characteristics of a userassociated with the plurality of ocular images, the characteristicsincluding at least one of age, genetics, sex, smoker, diet, healthparameters, concurrent illness and concurrent medications and therapies.40. The computer-implemented method of any one of claims 28 to 39,wherein the predicting tissue loss comprises predicting a rate of tissueloss.
 41. The computer-implemented method of any one of claims 28 to 40,wherein the predicting tissue loss comprises predicting whether tissueloss has previously occurred.
 42. The computer-implemented method of anyone of claims 28 to 41, wherein the predicting tissue loss comprisespredicting whether tissue loss will occur in the future.
 43. Thecomputer-implemented method of any one of claims 28 to 42, wherein thepredicting tissue loss comprises predicting regions of diseaseprogression and rate of disease progress.
 44. The computer-implementedmethod of any one of claims 28 to 43, wherein the predicting tissue losscomprises predicting progression from early to late dry Age RelatedMacular Degeneration (AMD).
 45. The computer-implemented method of anyone of claims 28 to 44, further comprising predicting a response of apatient to an intervention based at least in part on the features.
 46. Acomputer-implemented method for predicting neovascularization,comprising: receiving a plurality of ocular images corresponding to aplurality of imaging modalities; registering the plurality of ocularimages to a common coordinate system; classifying features of each ofthe plurality of ocular images using a pattern recognition model;predicting neovascularization based at least in part on the features.47. The computer-implemented method of claim 46, wherein the pluralityof imaging modalities comprises at least one of delayed near-infraredanalysis (DNIRA), infra-red reflectance (IR), confocal scanning laserophthalmoscopy (cSLO), fundus autofluorescence (FAF), color fundusphotography (CFP), optical coherence tomography (OCT), OCT-angiographyand fluorescence lifetime imaging (FLI).
 48. The computer-implementedmethod of claim 46 or 47, wherein the plurality of imaging modalitiesincluding cross-section images and en face images.
 49. Thecomputer-implemented method of any one of claims 46 to 48, wherein thepattern recognition model is a convolutional neural network built basedon training data corresponding to a plurality of ocular images andfeature extraction and feature selection performed to generate featuresfrom the training data.
 50. The computer-implemented method of claim 49,wherein the feature extraction generates a masked image of definedshapes.
 51. The computer-implemented method of claim 49 or 50, whereinthe feature selection comprises one or more of an are of intra-retinal,subretinal or sub-retinal pigment epithelium fluid using OCT, or dyeleakage using angiography.
 52. The computer-implemented method of anyone of claims 46 to 51, wherein the predicting neovascularization isbased on time series forecasting to predict tissue loss based on a timeseries data model.
 53. The computer-implemented method of claim 52,wherein the time series data model is generated based on multiplecross-section segmentation maps generated for each of multiple timepoints and corresponding cross-section images, each of the cross-sectionsegmentation maps corresponding to an en face region of an eye, and eachsegment of the cross-section segmentation map corresponding to across-section image at that region of the eye classified as a phenotypeof one of normal, drusen, retinal pigment epithelium detachment,geographic atrophy, macular atrophy or neovascularization, based atleast in part on classification of the cross-section image correspondingto that segment using a convolutional neural network.
 54. Thecomputer-implemented method of any one of claims 46 to 53, wherein thefeatures selected comprise phenotypes of a user associated with theplurality of ocular images.
 55. The computer-implemented method of claim54, further comprising identifying the phenotypes as risk factors bycorrelating the phenotypes with a rate of tissue loss over time.
 56. Thecomputer-implemented method of any one of claims 46 to 55, wherein thepredicting neovascularization is based at least in part on non-imagebased biomarker data.
 57. The computer-implemented method of claim 56,wherein the non-image based biomarker data comprises characteristics ofa user associated with the plurality of ocular images, thecharacteristics including at least one of age, genetics, sex, smoker,diet, health parameters, concurrent illness and concurrent medicationsor therapies.
 58. The computer-implemented method of any one of claims46 to 57, wherein the predicting neovascularization comprises predictinga onset of neovascularizaton.
 59. The computer-implemented method of anyone of claims 46 to 58, wherein the predicting neovascularizationcomprises predicting whether neovascularization has previously occurred.60. The computer-implemented method of any one of claims 46 to 59,wherein the predicting neovascularization comprises predicting whetherneovascularization will occur in the future.
 61. Thecomputer-implemented method of any one of claims 46 to 60, wherein thepredicting neovascularization comprises predicting regions of diseaseprogression and rate of disease progress.
 62. The computer-implementedmethod of any one of claims 46 to 61, wherein the predictingneovascularization comprises predicting progression from early to latedry Age Related Macular Degeneration (AMD).
 63. The computer-implementedmethod of any one of claims 46 to 62, further comprising predicting aresponse of a patient to an intervention based at least in part on thefeatures.